Hyperthreading DECREASES performance? There ought to be a ..

On 2009-10-28, Ignoramus27237 <ignoramus27237 RemoveThis @NOSPAM.27237.invalid> wrote:
> On 2009-10-28, Joe <joe RemoveThis @spam.hits-spam-buffalo.com> wrote:
>> And what DO you want? You are using a kernel that predates your CPU.
>> It is NOT optimized to run on your CPU, and will NOT run optimally.
>> If you'd like to test the performance, test it on a kernel that has
>> been built and optimized for the CPU you are running it on.
>
> OK, you convinced me. I will download and compile a new kernel,
> 2.6.31.5. It just might help. In fact I am already compiling it. It
> has hyperthreading support as a default option, which is encouraging.

Well, a new kernel did not help at all. Same problem.

I compiled it last night and tried at work today.

Linux version 2.6.31.5 (root@***) (gcc version 4.2.4 (Ubuntu 4.2.4-1ubuntu4)) #1 SMP Tue Oct 27 23:48:54 CDT 2009

No difference, same poor (and random) allocation of CPUs, with no HT
awareness.

I am not surprised about it because I think that these issues were
worked on years ago. These features either do not exist, or, more
likely, I do not know how to take advantage of them.

i

On 2009-10-28, The Natural Philosopher <tnp.TakeThisOut@invalid.invalid> wrote:
> Ignoramus27237 wrote:
>> On 2009-10-28, Joe <joe.TakeThisOut@spam.hits-spam-buffalo.com> wrote:
>>> On 2009-10-28, Ignoramus27237 <ignoramus27237.TakeThisOut@NOSPAM.27237.invalid> wrote:
>>>> On 2009-10-28, DanS <t.h.i.s.n.t.h.a.t.TakeThisOut@r.o.a.d.r.u.n.n.e.r.c.o.m> wrote:
>>>>> Ignoramus27237 <ignoramus27237.TakeThisOut@NOSPAM.27237.invalid> wrote in
>>>>> news:QtidnSTzfZ1mE3rXnZ2dnUVZ_tadnZ2d@giganews.com:
>>>>>
>>>>>>>> I wrote a test perl script,
>>>>>>> why would anyone use PERL for performance...
>>>>>>>
>>>>>> This was a test. The actual app is in C++. But a perl; script is a
>>>>>> much cleaner test because it is much simpler.
>>>>> Then it?s not very representative of the actual task.
>>>> It may not be representative of the actual task, but it is not very
>>>> far.
>>>>
>>>> I really do not want the discussion to stray into this area.
>>> And what DO you want? You are using a kernel that predates your CPU.
>>> It is NOT optimized to run on your CPU, and will NOT run optimally.
>>> If you'd like to test the performance, test it on a kernel that has
>>> been built and optimized for the CPU you are running it on.
>>
>> OK, you convinced me. I will download and compile a new kernel,
>> 2.6.31.5. It just might help. In fact I am already compiling it. It
>> has hyperthreading support as a default option, which is encouraging.
>>
>> i
> since that's where all the grunt work of multi-threading happens, that's
> probably a very good starting point

Nope, did not help, see my other post.

i

On 2009-10-28, The Natural Philosopher <tnp.RemoveThis@invalid.invalid> wrote:
>> I really do not want the discussion to stray into this area.
>
> but this is really the key.

No, it is not.

> multi-threading that you do not explicitly do yourself, is like e.g.
> memory management. The higher level language you use, the more likely it
> is that some arbitrary decisions by the language designers on OS
> designers will totally invalidate any real test you care to make.
>
> The fact that this does not seem to have occurred to you, is more
> worrying than you think.

I know enough about the app and the test to conclude that the test is
a very valid one.

I do not want to sound flippant about it, but I know what the app
does, how we manage threads, and what the test perl script does, to
realize that the perl script test is a pefect test for what I want to
find out.

What I do NOT know is how to make the scheduler allocate tasks to CPUs
for maximum performance.

i

Ignoramus27237 wrote:
> On 2009-10-28, The Natural Philosopher <tnp.TakeThisOut@invalid.invalid> wrote:
>> Ignoramus27237 wrote:
>>> At work, we have some users who ru a multithreaded app, and they need
>>> every single bit of performance we can squeeze from computers.
>>>
>>> We are looking into whether we can obtain additional speed performance
>>> from using Intel hyperthreading, as opposed to disabling
>>> hyperthreading.
>>>
>>> If we are able to get benefits from hyperthreading, it will be a huge
>>> argument towards converting certain high end users desktops to Linux
>>> from Windows XP.
>>>
>>> I wrote a test perl script,
>> why would anyone use PERL for performance...
>>
>
> This was a test. The actual app is in C++. But a perl; script is a
> much cleaner test because it is much simpler.
>

Well, at the machine code level, no, it isn't.

I don't claim to know much about multithreading, but I would expect to
be writing as close to the hardware as possible to extract benefits.

Possibly down to writing core routines in assembler to allocate tasks
amongst processors. Or using a pre made library with multi-threading
hooks in it.

Otherwise you have to accept the OS is going to second guess what to put
where, and that is in no way optimal for your application in all likelihood.


> i

Ignoramus27237 wrote:
> On 2009-10-28, DanS <t.h.i.s.n.t.h.a.t.TakeThisOut@r.o.a.d.r.u.n.n.e.r.c.o.m> wrote:
>> Ignoramus27237 <ignoramus27237.TakeThisOut@NOSPAM.27237.invalid> wrote in
>> news:QtidnSTzfZ1mE3rXnZ2dnUVZ_tadnZ2d@giganews.com:
>>
>>>>> I wrote a test perl script,
>>>> why would anyone use PERL for performance...
>>>>
>>> This was a test. The actual app is in C++. But a perl; script is a
>>> much cleaner test because it is much simpler.
>> Then it?s not very representative of the actual task.
>
> It may not be representative of the actual task, but it is not very
> far.
>
> I really do not want the discussion to stray into this area.
>

but this is really the key.

multi-threading that you do not explicitly do yourself, is like e.g.
memory management. The higher level language you use, the more likely it
is that some arbitrary decisions by the language designers on OS
designers will totally invalidate any real test you care to make.

The fact that this does not seem to have occurred to you, is more
worrying than you think.


> i

Ignoramus27237 wrote:
> On 2009-10-28, Joe <joe RemoveThis @spam.hits-spam-buffalo.com> wrote:
>> On 2009-10-28, Ignoramus27237 <ignoramus27237 RemoveThis @NOSPAM.27237.invalid> wrote:
>>> On 2009-10-28, DanS <t.h.i.s.n.t.h.a.t RemoveThis @r.o.a.d.r.u.n.n.e.r.c.o.m> wrote:
>>>> Ignoramus27237 <ignoramus27237 RemoveThis @NOSPAM.27237.invalid> wrote in
>>>> news:QtidnSTzfZ1mE3rXnZ2dnUVZ_tadnZ2d@giganews.com:
>>>>
>>>>>>> I wrote a test perl script,
>>>>>> why would anyone use PERL for performance...
>>>>>>
>>>>> This was a test. The actual app is in C++. But a perl; script is a
>>>>> much cleaner test because it is much simpler.
>>>> Then it?s not very representative of the actual task.
>>> It may not be representative of the actual task, but it is not very
>>> far.
>>>
>>> I really do not want the discussion to stray into this area.
>> And what DO you want? You are using a kernel that predates your CPU.
>> It is NOT optimized to run on your CPU, and will NOT run optimally.
>> If you'd like to test the performance, test it on a kernel that has
>> been built and optimized for the CPU you are running it on.
>
> OK, you convinced me. I will download and compile a new kernel,
> 2.6.31.5. It just might help. In fact I am already compiling it. It
> has hyperthreading support as a default option, which is encouraging.
>
> i
since that's where all the grunt work of multi-threading happens, that's
probably a very good starting point

In <3--dnTZSge6r53rXnZ2dnUVZ8jidnZ2d.RemoveThis@pipex.net>, on 10/27/2009
at 10:54 PM, "Brian Gregory [UK]" <ng.RemoveThis@bgdsv.co.uk> said:

>One would hope that on any system with hyperthreading processors the
>cores would all be busy with one thread before any would be asked to
>execute a second thread. It seems like common sense to me and I'd be
>amazed if any operating system did differently.

The issue is more complicated than it might appear at first lance. The
dispatcher must attempt to satisfy two conflicting requirements.

1. Balance the load. That requires scattering threads evenly across
chips, cores and virtual cores.

2. Maximize cache efficiency. That requires dispatching related
threads to the same core; failing that, to the same chip.

Also, keep in mind that the relative importance of the issues will depend
on whether the threads are memory bound or processor bound.

--
Shmuel (Seymour J.) Metz, SysProg and JOAT <http://patriot.net/~shmuel>

Unsolicited bulk E-mail subject to legal action. I reserve the
right to publicly post or ridicule any abusive E-mail. Reply to
domain Patriot dot net user shmuel+news to contact me. Do not
reply to spamtrap.RemoveThis@library.lspace.org

Am Mittwoch, 28. Oktober 2009 15:04, Ignoramus8868 a écrit :

> I am not surprised about it because I think that these issues were
> worked on years ago. These features either do not exist, or, more
> likely, I do not know how to take advantage of them.

man 2 sched_setaffinity gettid

--
printf -v email $(echo \ 155 141 162 143 145 154 142 162 165 151 \
156 163 155 141 100 171 141 150 157 157 056 143 157 155|tr \ \\\\)
# Live every life as if it were your last! #

On 2009-10-28, Shmuel Metz <spamtrap.RemoveThis@library.lspace.org.invalid> wrote:
> In <3--dnTZSge6r53rXnZ2dnUVZ8jidnZ2d.RemoveThis@pipex.net>, on 10/27/2009
> at 10:54 PM, "Brian Gregory [UK]" <ng.RemoveThis@bgdsv.co.uk> said:
>
>>One would hope that on any system with hyperthreading processors the
>>cores would all be busy with one thread before any would be asked to
>>execute a second thread. It seems like common sense to me and I'd be
>>amazed if any operating system did differently.
>
> The issue is more complicated than it might appear at first lance. The
> dispatcher must attempt to satisfy two conflicting requirements.
>
> 1. Balance the load. That requires scattering threads evenly across
> chips, cores and virtual cores.
>
> 2. Maximize cache efficiency. That requires dispatching related
> threads to the same core; failing that, to the same chip.
>
> Also, keep in mind that the relative importance of the issues will depend
> on whether the threads are memory bound or processor bound.

Great point.

Based on what I know about the app, which I (among others) developed,
the threads _mostly_ access their own memory (they do not share that
much) and do heavy computations.

With this in mind, I would give higher importance to your item 1.

The test with four parallel perl script instances, was a perfect
example of where item 1 was more important, if for not other reason
that they were separate processes.

Also note that the allocation of processes to virtual CPUs, seems to
be done completely randomly, without a set pattern.

You sound like you know a lot about this stuff. If so, do you know if
there is some option I can turn on, to make sure that processes (or
threads) would preferably be scattered around different cores first.

I found an Intel article that I mentioned earlier, that talks about
hyperthreading and linux scheduler, as if this was a done deal.

http://software.intel.com/sites/oss/pdfs/mclinux.pdf

However, that I do not see the outcome that the article would imply.

i

On 10/27/2009 03:31 PM, Ignoramus27237 wrote:
....
> The results were actually a disappointment, if the number of tasks was
> equal to the number of physical cores. For the test with four parallel
> subprocesses, on four CPUs, It takes longer to run it with HT than
> without HT.

Sorry, no answers but, I remember this being an issue in FreeBSD for a
number of years. AFAIK, two things have changed the situation for the
better: they rewrote ULE from a "Giant Lock" scheduler to a more
granular locking mechanism in circa FreeBSD 6. Also, apparently HTT is
substantially reworked in Intel's new i7 line.

There's some discussion & testing of it by a gent named Graaf in the
FreeBSD forums at:

<http://forums.freebsd.org/showthread.php?t=2561>
<http://forums.freebsd.org/showthread.php?t=3185>

fwiw,
--
-Craig

On Tue, 27 Oct 2009 17:31:58 -0500, Ignoramus27237 wrote:

>
> At work, we have some users who ru a multithreaded app, and they need
> every single bit of performance we can squeeze from computers.
>
>
<snip>
>
> This is Ubuntu Hardy, 2.6.24 kernel.
>
> Thanks
>
IME, the 2.6.2x kernel series performance was all over the map*. IMO,
you'd be better off investigating the "latest and greatest," beginning
with a new kernel in the 2.6.3x series. This is especially true if you
are truly seeking the best performance. The kernel's changelog shows major
modifications with respect to its scheduler, block io, and filesystem
subsystems.

* One thing I definitely observed is a dual Pentium III 1.26 GHz
motherboard with 1.5G RAM outperform a Core 2 Duo motherboard with an
E6750 CPU with 2.0G RAM. This was observed mostly with virtualization
tasks. Common sense says that should not happen, and it is pointing to
something seriously amiss with the scheduler and/or "sleep" states. My
guess is that so much effort was put into achieving deep sleep states that
the CPU appears to be idle most of the time, even when it should be
working. My WAG with respect to the earlier CPU: it did not exhibit the
flaw because it does not have "sleep" states.

--
Douglas Mayne

Ignoramus8868 a écrit :

>
> What I do NOT know is how to make the scheduler allocate tasks to CPUs
> for maximum performance.

apt-get install schedutils

then

man taskset

On 2009-10-28, GuiGui <GuiGui.RemoveThis@nospam.fr> wrote:
> Ignoramus8868 a ?crit :
>
>>
>> What I do NOT know is how to make the scheduler allocate tasks to CPUs
>> for maximum performance.
>
> apt-get install schedutils
>
> then
>
> man taskset

OK, it was very interesting. taskset was in utils-linux actually, but
that does not matter.

I did indeed verify that with "taskset", if I give it mask 55, to use
only even numbered CPUs, the tasks of my test script would be
perfectly allocated and would all complete quickly.

It was very educational to try it out.

However, I would like the scheduler to make this decision, instead of
me specifying, for a very simple reason. In the actual app, the
threads run at random moments in a random quantity. So I would want
the scheduler to PREFER to use different physical cores, if possible,
without me babysitting it and specifying where to run each task or
thread.

That's what that Intel article that I posted, talks about, but I
cannot seem to get that behavior.

i