thread memory size

Rainer Weikusat wrote:
>
> That's your merciful assumption :->. I am fairly convinced that this
> is part of the stern wave of this
>
> http://www.barrelfish.org/
>

You lost me now, but thats ok. Because I found an article stating how
threads and kernel memory work. Linux don't have its own threads
structure its only got something called a task_structure, which is
treated equally to processes by the scheduler. The only difference is
that the task structure is part of an existing process, so the task
structure does not need to contain too much data, as all the rest is
properly set up by the process structures. That mean a thread in linux
is quite lightweight and it also explains why the new thread
implementation in Linux can start 200K threads in 2 seconds. Which I am
happy about, now I just need to figure out how to configure Linux to be
able to run that many threads So now I can start testing this kind of
massive parallelism on a Linux machine, of course not with 1 million
threads, yet. But I can have some fun with it.

(It interesting how I was only asking how much kernel memory linux
requires per thread. And as any person discussing on Usenet, I will
argue my case. But for some reason this discussion has shifted into a
completely new theme, which is some sort of bashing against what you
don't like or believe in. I have no problems with what you believe in or
like. I only ask that you answer my questions in a serious matter,
instead of trying to belittle things that you don't agree with. The
world was once flat...
(I am not saying I am Columbus nor Copernicus, I am just saying you dont
know everything and should therefore give ideas some reasonable amount
respect until you do understand it.))

regards

Jan

Jan Helgesen <spam RemoveThis @nospam.com> writes:
> Rainer Weikusat wrote:
>> That's your merciful assumption :->. I am fairly convinced that this
>> is part of the stern wave of this
>>
>> http://www.barrelfish.org/
>>
>
> You lost me now, but thats ok.

The time coincedence is striking.

> Because I found an article stating how threads and kernel memory
> work. Linux don't have its own threads structure its only got
> something called a task_structure, which is treated equally to
> processes by the scheduler. The only difference is that the task
> structure is part of an existing process, so the task structure does
> not need to contain too much data, as all the rest is properly set
> up by the process structures.

There is no 'process structure'. The kernel scheduler deals with
'tasks', as represtented by a struct task_struct and these 'tasks' may
share all kinds of resources, eg open files tables, address spaces,
filesystem namespace etc, see clone( for details. Depending on what
is and what isn't shared, 'tasks' can be used to provide both POSIX
threads and UNIX(*) processes.

[...]

> (It interesting how I was only asking how much kernel memory linux
> requires per thread.

And I had answered this question, hadn't you immediatly started to try
to lecture me about 'proper terms' based on your (almost complete)
lack of knowledge in this area. You still haven't found either the
amount of kernel memory used by a thread (which is mostly the
associated kernel stack, default 8K, configurable 4K) nor the problem
with your other limit: the virtual address space. Each thread needs a
separate part of that for its stack.

> And as any person discussing on Usenet, I will
> argue my case. But for some reason this discussion has shifted into a
> completely new theme, which is some sort of bashing against what you
> don't like or believe in.

This is called 'argueing against something', eg your absolutely
ridicolous idea of the 256-way-SMP-problem or your misconception about
inter-thread communication (which is usually done in exactll the way
you desribed as 'new idea people would resent for irrational
reasons'). You are, of course, not even trying to provide counter
arguments (and given that you were simply wrong in both the cases I
mentioned, that would be somewhat difficult), instead you chose to
bash people (me, in this case) based on my style of writing.

> I have no problems with what you believe in
> or like. I only ask that you answer my questions in a serious matter,
> instead of trying to belittle things that you don't agree with. The
> world was once flat...

Your world is still flat. BTW, please stop this 'posted and mailed'
nonsense.

Ok, so if you think I have misunderstood it all and I barely have any
knowledge about anything, I will give you a chance to explain,
preferably in different wording than you already have, what you think I
am misunderstanding so completely.

regards

Jan

Jan Helgesen <spam.RemoveThis@nospam.com> writes:
> Ok, so if you think I have misunderstood it all and I barely have any
> knowledge about anything, I will give you a chance to explain,
> preferably in different wording than you already have, what you think
> I am misunderstanding so completely.

Since you have again sent me an e-mail from your fake address and have
(apparently) completely ignored the other parts of my posting, namely,
the technical explanations, as well, why would I repeat myself? You
aren't listening, anyway.

On Oct 6, 5:37 am, Jan Helgesen <s....RemoveThis@nospam.com> wrote:

> David Schwartz wrote:

> > What he's missing is that the whole point of limiting all your
> > inter-'thread' communication to message passing is that your threads
> > then don't have to share a vm. So he's trying to add message passing
> > to threads that share a vm and imagine scaling that to thousands of
> > cores.

> > This totally defeats the point of message passing.

> Its irrelevant how the message passing is implemented, as long as its
> efficient enough and supports the properties the system design requires.
> Everything else is just technical excuses used by proponents of a
> certain technology or mindset.

I agree. But please show some sign that you understand what I'm
saying.

> Just to make this clear when I say Shared Memory message passing I am
> not talking about OS Shared Memory. I am talking about process memory
> that is shared between threads.

What do you think is the difference between these two things?

> Further on if this had been a question of IPC, we would be talking about
> processes. But processes are too heavywight to be able to run on a
> machine with 1 million cores. To be able to do that one need ultra
> lightweight execution structures. It does not matter whether it is
> called a process or a thread, as long as it is much lighter than it is
> today and hence does not cost too much in memory or kernel synchronisation.
> the question of how the message passing is implemented does no matter,
> because what one wants is not to keep data in one place for all to
> access, but to send the data around the program so that only one
> thread/process has the ability to access the data, thus eliminating
> locks and synchronisation issues.

I agree.

So, do you understand any of what I said?

I will try it one more time:

1) If you have threads that share all their memory, this shared memory
will be the fastest IPC you have. There would be no benefit to using
messages for IPC. That would just be an extra layer of overhead.

2) It takes effort to use messages instead of shared memory. Things
have to be packaged and unpacked.

3) This effort has a huge payoff though -- your threads don't have to
share all their vm. This is good because shared vm has a high cost.

4) Attempting to use a message passing layer on top of a shared vm
threads architecture would be very, very sub-optimal. You'd pay all
the costs of shared vm and get none of the benefits.

5) So if someone were attempting to create a message-passing
architecture that was intended to be highly efficient and scalable,
the last thing they would do is start with threads that share a vm.
Nobody would ever do that, it would be complete idiocy.

If someone were designing a high-performance, message-passing
concurrency platform intended to scale to thousands of cores, they
would *NOT* ever, under any circumstances start with shared-vm
threads.

Your question is like asking how many miles per gallon a Volvo wagon
gets because you're considering adapting it to fly tons of cargo
across the country.

DS

Rainer Weikusat wrote:
> Jan Helgesen <spam RemoveThis @nospam.com> writes:
>> Ok, so if you think I have misunderstood it all and I barely have any
>> knowledge about anything, I will give you a chance to explain,
>> preferably in different wording than you already have, what you think
>> I am misunderstanding so completely.
>
> Since you have again sent me an e-mail from your fake address and have
> (apparently) completely ignored the other parts of my posting, namely,
> the technical explanations, as well, why would I repeat myself? You
> aren't listening, anyway.

That's just an pitiful excuse. If you have a compelling argument then
spit it out. Don't be a jerk and try to pretend you know something when
you don't want to share it. I am listening if you can be bothered to say
something without shouting, screaming and mocking people. If not then I
can only remain with my current conclusion of your arguments.

regards

Jan

On 10/06/2009 10:25 AM, Jan Helgesen wrote:
> Rainer Weikusat wrote:
>> Jan Helgesen <spam DeleteThis @nospam.com> writes:
>>> Ok, so if you think I have misunderstood it all and I barely have any
>>> knowledge about anything, I will give you a chance to explain,
>>> preferably in different wording than you already have, what you think
>>> I am misunderstanding so completely.
>>
>> Since you have again sent me an e-mail from your fake address and have
>> (apparently) completely ignored the other parts of my posting, namely,
>> the technical explanations, as well, why would I repeat myself? You
>> aren't listening, anyway.
>
> That's just an pitiful excuse. If you have a compelling argument then
> spit it out.

He has. Go back and read the information in this thread. The answer to
your question (as well as several corrections to your misconceptions and
the introduction of several issues which you likely haven't considered)
has already been covered.

Chris

On Oct 6, 4:38 pm, Jan Helgesen <s....RemoveThis@nospam.com> wrote:

> > 5) So if someone were attempting to create a message-passing
> > architecture that was intended to be highly efficient and scalable,
> > the last thing they would do is start with threads that share a vm.
> > Nobody would ever do that, it would be complete idiocy.

> Now this makes me wonder if we are talking about the same thing.
> VM is virtual memory. Virtual memory is the processes complete address
> space. Per definition a thread is sharing its memory with other threads
> in the same memory space, i.e. virtual memory address space.
> So you statement "... start with threads that share a vm", is
> nonsensical because all threads must share vm, by definition.
> But they don't have to share their local variables.

It's not nonsensical. If you're going to limit your IPC to message
passing to get scalability, the reason you would get scalability would
be that you wouldn't have a shared vm. That's where the scalability
would come from. The whole point using message passing as an IPC to
get around the scalability limitation of shared vm.

Yes, it also may have some other benefits, but the *scalability*
benefit is that you don't have to share a vm.

> > If someone were designing a high-performance, message-passing
> > concurrency platform intended to scale to thousands of cores, they
> > would *NOT* ever, under any circumstances start with shared-vm
> > threads.

> If on the other hand you are saying, that since threads already share
> address space using shared memory is unnecessary, then I agree.
> But as I explained above, I am not talking about the OS Shared memory.

That's not what I'm saying. Please read exactly what I said word-for-
word. Here it is again:

"If someone were designing a high-performance, message-passing
concurrency platform intended to scale to thousands of cores, they
would *NOT* ever, under any circumstances start with shared-vm
threads."

And the reason is simple:

1) They need performance.
2) They don't need shared vm.
3) Shared vm has a high performance cost.

So your question is about how to modify a Volkswagen to fly freight.
You *don't* start with a Volkswagen to fly freight, that's not a
sensible question.

DS

David Schwartz wrote:
> On Oct 6, 5:37 am, Jan Helgesen <s....DeleteThis@nospam.com> wrote:
>
>> David Schwartz wrote:
>
>>> What he's missing is that the whole point of limiting all your
>>> inter-'thread' communication to message passing is that your threads
>>> then don't have to share a vm. So he's trying to add message passing
>>> to threads that share a vm and imagine scaling that to thousands of
>>> cores.
>
>>> This totally defeats the point of message passing.
>
>> Its irrelevant how the message passing is implemented, as long as its
>> efficient enough and supports the properties the system design requires.
>> Everything else is just technical excuses used by proponents of a
>> certain technology or mindset.
>
> I agree. But please show some sign that you understand what I'm
> saying.
>
>> Just to make this clear when I say Shared Memory message passing I am
>> not talking about OS Shared Memory. I am talking about process memory
>> that is shared between threads.
>
> What do you think is the difference between these two things?

Ah, I see where the confusion is. I did not word it correctly. Let me
try again:

First of all, there are two shared memory principles:

1- OS Shared memory: which is the IPC mechanism called Shared Memory,
also known as System V IPC or BSD mmap. This is an inter *process*
communication technique.
2- Shared memory programming model: which is using the global address
space available within a process. It is accessible to the process and
its threads. This is also called a programs global memory.

Now here is where I did not keep my head on straight. I have been
talking about Shared memory message passing, but what I meant to say was
that there is a difference between

- the above mentioned Shared memory programming model and
- message passing programming model.

the shared memory programming model uses global memory with locks and
synchronisation for concurrency control while
the message passing model is copying the data from a threads private
variables to another threads private variables by way of a pipe.

Rudimentary the message passing model could be done by only allowing a
thread to use local variables for data storage and a pipe to send
messages with the data to another thread. This would probably be the way
to do it on the sequential computers of today

A system designed for super massive parallelism (at least thousands of
cores) with support for the message passing model instead of shared
memory model would perhaps use a different scheme, which might be more
efficient and less cumbersome. It depends.

So, my main point was that using the memory passing programming model
instead of shared memory programming model, one would not have problems
with concurrency programming usually connected to locks and synchronisation.

Now, someone argued that there would still be locks and synchronisation,
because that would be needed in the pipe. And that is true, but as I
have already said, the concurrency model for a pipe is so much simpler
than the concurrency model for a random global variable in a program.
Also the pipe would be implemented once and tested time and time again,
with little change in code. A random global variable for a random
program would be the Nth invention of the wheel, done by a new
programmer or used in a new context, which now needs to review his
concurrency model yet again to check that there are no deadlocks,
livelocks, starvations and so on.


For the rest of your questions, I have written an answer, but your
questions might not be as interesting any more since they where based on
a wrong statement.


> So, do you understand any of what I said?
>
> I will try it one more time:
>
> 1) If you have threads that share all their memory, this shared memory
> will be the fastest IPC you have. There would be no benefit to using
> messages for IPC. That would just be an extra layer of overhead.

If the only requirement are that it happens as fast as possibly, then
you are right. Fast is not the only requirement when creating a program.
Other requirements are functional correctness, stability, error
management, complexity management, modularisation, maintainability and
so on.

But to answer your question more directly. Using OS shared memory to
build a message passing pipe on top of it makes little sense in it self.
Because that is putting one IPC on top of another. But then we go back
to what I wrote in the section above this.

>
> 2) It takes effort to use messages instead of shared memory. Things
> have to be packaged and unpacked.

Yes, that is true, but it depends on how its packaged and that is an
implementational detail of the pipe technology. Albeit, with ordinary
shared memory or with function calls, packaging are also done. Because
the data have to fit the arguments required by the structure of the
shared memory or the function call declaration.
It would be utterly impractical to use e.g. XML for packaging this kind
of message passing.

I wont comment on you point 3 and 4 as I believe I have already answered
that.

> 5) So if someone were attempting to create a message-passing
> architecture that was intended to be highly efficient and scalable,
> the last thing they would do is start with threads that share a vm.
> Nobody would ever do that, it would be complete idiocy.

Now this makes me wonder if we are talking about the same thing.
VM is virtual memory. Virtual memory is the processes complete address
space. Per definition a thread is sharing its memory with other threads
in the same memory space, i.e. virtual memory address space.
So you statement "... start with threads that share a vm", is
nonsensical because all threads must share vm, by definition.
But they don't have to share their local variables.

> If someone were designing a high-performance, message-passing
> concurrency platform intended to scale to thousands of cores, they
> would *NOT* ever, under any circumstances start with shared-vm
> threads.

If on the other hand you are saying, that since threads already share
address space using shared memory is unnecessary, then I agree.
But as I explained above, I am not talking about the OS Shared memory.

regards

Jan

Chris Friesen wrote:
>
> He has. Go back and read the information in this thread. The answer to
> your question (as well as several corrections to your misconceptions and
> the introduction of several issues which you likely haven't considered)
> has already been covered.

I disagree, the only answer he has given was where he stated that the
task_struct is used by both a thread and a process. He says that there
is no process structure. But a task_struct is a process structure by any
other name. Such a structure contains process scheduler informations (
state, priority, schedler_class etc), and also contains information such
as policy, cpus_allowed, pid, ppid, cpu time and also a pointe to the
stack), as he pointed out. But this structure takes up memory. And that
is where my question was. This is memory which would not be occupied if
the thread or process was not started.

So the fact that Linux shares that structure between both processes and
threads, has very little implications for my question. My question was
how big is that structure, and his answer had little to do with that.
(a proper answer to those questions might be found in my last answer to
Schwartz)
Weikusat did give answers to a lot of things, which was a digression
from the question I asked.

And then, one of his last statements was a rhetorical statement that I
was no closer to my getting an answer. One possible reason for that is
the he offered me no answer to the question I asked.

So now I have managed to find the structure in the linux code and see
that its not as lightweight as I exclaimed it might be a couple of posts
ago.


regards

Jan

David Schwartz wrote:
>
> It's not nonsensical. If you're going to limit your IPC to message
> passing to get scalability, the reason you would get scalability would
> be that you wouldn't have a shared vm. That's where the scalability
> would come from. The whole point using message passing as an IPC to
> get around the scalability limitation of shared vm.
>
> Yes, it also may have some other benefits, but the *scalability*
> benefit is that you don't have to share a vm.

Yes this is what I am saying, but with more correct wording.

Just to make it clear, I understand perfectly what you are trying to
say. But what the words you are using are not correct for that
statement, so it causes confusion.

A thread is per definition required to share vm. You cant define your
way out of that. If you are talking about something that does not share
vm then you are either talking about processes or you are talking about
some completely new execution unit which no-one has hardly ever heard of
before. Do you want to call it something else, say ultra flyweight
processes perhaps?

>
> That's not what I'm saying. Please read exactly what I said word-for-
> word. Here it is again:
>
> "If someone were designing a high-performance, message-passing
> concurrency platform intended to scale to thousands of cores, they
> would *NOT* ever, under any circumstances start with shared-vm
> threads."

As I stated, threads are per definition shared-vm.

>
> And the reason is simple:
>
> 1) They need performance.

That is correct

> 2) They don't need shared vm.

Well, i have said it two times already...

> 3) Shared vm has a high performance cost.

what cost is that, and the high performance cost is compared to what?

regards

Jan

On Oct 6, 5:32 pm, Jan Helgesen <s....DeleteThis@nospam.com> wrote:

> > 3) Shared vm has a high performance cost.

> what cost is that, and the high performance cost is compared to what?

Anything that manipulates the vm has to be synchronized. Consider, for
example, opening a file. Consider mapping a new region of memory.
Also, since they have to be in the same SMP region, they typically
have to all be talking to a single kernel. So many kernel operations
will wind up imposing some kind of synchronization cost.

The whole point of message passing for scalability is that you don't
have to use a shared vm. You can distribute your large operation over
multiple memory domains. Shared vm among thousands of cores make
absolutely no sense at all for the foreseeable future.

DS

On Oct 7, 12:24 am, Jan Helgesen <s... RemoveThis @nospam.com> wrote:

> > The whole point of message passing for scalability is that you don't
> > have to use a shared vm. You can distribute your large operation over
> > multiple memory domains. Shared vm among thousands of cores make
> > absolutely no sense at all for the foreseeable future.

> I am having problems understanding what you are saying here.

Apparently.

> First off, when you say vm, you mean virtual memory, right?
> Wikipedia defines virtual memory as:

Yes.

> "Virtual memory is a computer system technique which gives an
> application program the impression that it has contiguous working memory
> (an address space), while in fact it may be physically fragmented and
> may even overflow on to disk storage."

> So virtual memory is not a process or programs complete address space,

Umm, yes it is. The process or program's complete address space is its
vm. (Perhaps you're not familiar with this usage, but it's very
common.)

> it is the technique that the kernel uses to let the process believe it
> has got a contiguous address space, agreed?

That's one of the things it does.

> So the question is then, when you say "shared vm", do you mean

> 1) shared memory (as in the IPC technique called System V Shared Memory)

No. In that case, each process has its own view of the memory. The vm
is not shared, the pages are shared.

> 2) the processes complete address space
>
> I will get back to answering the other part of the post after you have
> given me an answer.

The term 'vm' means the process' view of memory -- memory as seen by
the process. Each process has its own vm. Threads have a single shared
vm.

DS