Quantcast

[Jack-Devel] jackdbus process eats two CPU cores for hours on end

classic Classic list List threaded Threaded
15 messages Options
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

[Jack-Devel] jackdbus process eats two CPU cores for hours on end

Thomas Howe
Hi all,

I'm having a problem with the jackdbus process; after some hours it starts maxing out two of my four CPU cores, which in turn causes xruns when running apps. It looks like other people have found the same problem, but no solutions.

I cut the important bits out and sped it up. It shows me starting jack via QJackCtl with a monitor of the log file and QJackCtl's messages in the first section, the processor spike in the second, an xrun in the third, and the processors returning to normal in the fourth. There's a clock at the bottom left corner of the screen.

I'm running Arch with the Linux 4-8-4.1-ARCH (x86_64) kernel. Let me know what other information would be useful! I can't figure out how to even start debugging this.

Thanks for watching,
Tom

_______________________________________________
Jack-Devel mailing list
[hidden email]
http://lists.jackaudio.org/listinfo.cgi/jack-devel-jackaudio.org
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: jackdbus process eats two CPU cores for hours on end

Markus Seeber
On 12/04/2016 05:26 AM, Thomas Howe wrote:

> Hi all,
>
> I'm having a problem with the jackdbus process; after some hours it starts
> maxing out two of my four CPU cores, which in turn causes xruns when
> running apps. It looks like other people have found the same problem, but
> no solutions.
>
> Here's a video of this happening:
> https://drive.google.com/open?id=0B2eROo5JatB4eHZweUtROTduSWc
> I cut the important bits out and sped it up. It shows me starting jack via
> QJackCtl with a monitor of the log file and QJackCtl's messages in the
> first section, the processor spike in the second, an xrun in the third, and
> the processors returning to normal in the fourth. There's a clock at the
> bottom left corner of the screen.
>
> I'm running Arch with the Linux 4-8-4.1-ARCH (x86_64) kernel. Let me know
> what other information would be useful! I can't figure out how to even
> start debugging this.
>
> Thanks for watching,
> Tom
>

Did this happen once, or does that happen always after the same amount
of time?
_______________________________________________
Jack-Devel mailing list
[hidden email]
http://lists.jackaudio.org/listinfo.cgi/jack-devel-jackaudio.org
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: jackdbus process eats two CPU cores for hours on end

Harry van Haaren
In reply to this post by Thomas Howe
On Sun, Dec 4, 2016 at 4:26 AM, Thomas Howe <[hidden email]> wrote:
I'm having a problem with the jackdbus process; after some hours it starts maxing out two of my four CPU cores, which in turn causes xruns when running apps. It looks like other people have found the same problem, but no solutions.


I could well be wrong; but my suspicion is not on audio threads - but rather management threads for things like DBus integration, or other "non realtime" operations. If it was the audio thread that was causing the deadlock and spinning, you would hear *no* audio at all.

So next steps I'd suggest running without any extra features (disable Dbus support, or Pulseaudio integration, or ALSA loopback devices, or anything else that's not "core" JACK). See if it still happens..
 
Good luck! -Harry

--

_______________________________________________
Jack-Devel mailing list
[hidden email]
http://lists.jackaudio.org/listinfo.cgi/jack-devel-jackaudio.org
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: jackdbus process eats two CPU cores for hours on end

Thomas Howe
I haven't measure how long it takes to happen, but it tends to be 'after a while'. This is the first time I've realised it stops after several more hours, which was a surprise. I'm trying to measure how long it takes now.

Is there a way to figure out what's getting processed when this happens?

I'm currently running the process with both of QJackCtl's d-bus boxes unticked to see if (and when) it happens again. PulseAudio isn't installed and I didn't have my ALSA bridge running when recording the video, but I won't start it today anyway just in case it would make a difference. No process spike yet...

On 4 December 2016 at 11:01, Harry van Haaren <[hidden email]> wrote:
On Sun, Dec 4, 2016 at 4:26 AM, Thomas Howe <[hidden email]> wrote:
I'm having a problem with the jackdbus process; after some hours it starts maxing out two of my four CPU cores, which in turn causes xruns when running apps. It looks like other people have found the same problem, but no solutions.


I could well be wrong; but my suspicion is not on audio threads - but rather management threads for things like DBus integration, or other "non realtime" operations. If it was the audio thread that was causing the deadlock and spinning, you would hear *no* audio at all.

So next steps I'd suggest running without any extra features (disable Dbus support, or Pulseaudio integration, or ALSA loopback devices, or anything else that's not "core" JACK). See if it still happens..
 
Good luck! -Harry

--


_______________________________________________
Jack-Devel mailing list
[hidden email]
http://lists.jackaudio.org/listinfo.cgi/jack-devel-jackaudio.org
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: jackdbus process eats two CPU cores for hours on end

Thomas Howe
It happens without dbus support (although it is still jackdbus installed).

This time it lasted between 6 hours and 6 hours 30 minutes, and the video shows between 6 hours 12 minutes and 6 hours 13 minutes, so I'm guessing it's about the same duration each time before it spikes, but I'd need to verify this to be sure.

I guess I'll try the jack2 package instead. Any ideas on where I should report this bug?

On 4 December 2016 at 20:43, Thomas Howe <[hidden email]> wrote:
I haven't measure how long it takes to happen, but it tends to be 'after a while'. This is the first time I've realised it stops after several more hours, which was a surprise. I'm trying to measure how long it takes now.

Is there a way to figure out what's getting processed when this happens?

I'm currently running the process with both of QJackCtl's d-bus boxes unticked to see if (and when) it happens again. PulseAudio isn't installed and I didn't have my ALSA bridge running when recording the video, but I won't start it today anyway just in case it would make a difference. No process spike yet...


On 4 December 2016 at 11:01, Harry van Haaren <[hidden email]> wrote:
On Sun, Dec 4, 2016 at 4:26 AM, Thomas Howe <[hidden email]> wrote:
I'm having a problem with the jackdbus process; after some hours it starts maxing out two of my four CPU cores, which in turn causes xruns when running apps. It looks like other people have found the same problem, but no solutions.


I could well be wrong; but my suspicion is not on audio threads - but rather management threads for things like DBus integration, or other "non realtime" operations. If it was the audio thread that was causing the deadlock and spinning, you would hear *no* audio at all.

So next steps I'd suggest running without any extra features (disable Dbus support, or Pulseaudio integration, or ALSA loopback devices, or anything else that's not "core" JACK). See if it still happens..
 
Good luck! -Harry

--



_______________________________________________
Jack-Devel mailing list
[hidden email]
http://lists.jackaudio.org/listinfo.cgi/jack-devel-jackaudio.org
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: jackdbus process eats two CPU cores for hours on end

John Rigg-16
In reply to this post by Thomas Howe
On Sun, Dec 04, 2016 at 04:26:14AM +0000, Thomas Howe wrote:

> Hi all,
>
> I'm having a problem with the jackdbus process; after some hours it starts
> maxing out two of my four CPU cores, which in turn causes xruns when
> running apps. It looks like other people have found the same problem, but
> no solutions.
>
> Here's a video of this happening:
> https://drive.google.com/open?id=0B2eROo5JatB4eHZweUtROTduSWc
> I cut the important bits out and sped it up. It shows me starting jack via
> QJackCtl with a monitor of the log file and QJackCtl's messages in the
> first section, the processor spike in the second, an xrun in the third, and
> the processors returning to normal in the fourth. There's a clock at the
> bottom left corner of the screen.
>
> I'm running Arch with the Linux 4-8-4.1-ARCH (x86_64) kernel. Let me know
> what other information would be useful! I can't figure out how to even
> start debugging this.

I couldn't watch the video here ('format not supported' error), but for
how long and at what sample rate are you trying to run JACK? IIRC
jack transport stores sample frame count as a 32-bit unsigned integer.
At 48kHz that's a little over 24hr before it hits the end of the counter.

John
_______________________________________________
Jack-Devel mailing list
[hidden email]
http://lists.jackaudio.org/listinfo.cgi/jack-devel-jackaudio.org
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: jackdbus process eats two CPU cores for hours on end

Stéphane Letz
In reply to this post by Thomas Howe
You should try to find out which thread eats the CPU.

Stéphane

> Le 5 déc. 2016 à 01:34, Thomas Howe <[hidden email]> a écrit :
>
> It happens without dbus support (although it is still jackdbus installed).
>
> This time it lasted between 6 hours and 6 hours 30 minutes, and the video shows between 6 hours 12 minutes and 6 hours 13 minutes, so I'm guessing it's about the same duration each time before it spikes, but I'd need to verify this to be sure.
>
> I guess I'll try the jack2 package instead. Any ideas on where I should report this bug?
>
> On 4 December 2016 at 20:43, Thomas Howe <[hidden email]> wrote:
> I haven't measure how long it takes to happen, but it tends to be 'after a while'. This is the first time I've realised it stops after several more hours, which was a surprise. I'm trying to measure how long it takes now.
>
> Is there a way to figure out what's getting processed when this happens?
>
> I'm currently running the process with both of QJackCtl's d-bus boxes unticked to see if (and when) it happens again. PulseAudio isn't installed and I didn't have my ALSA bridge running when recording the video, but I won't start it today anyway just in case it would make a difference. No process spike yet...
>
>
> On 4 December 2016 at 11:01, Harry van Haaren <[hidden email]> wrote:
> On Sun, Dec 4, 2016 at 4:26 AM, Thomas Howe <[hidden email]> wrote:
> I'm having a problem with the jackdbus process; after some hours it starts maxing out two of my four CPU cores, which in turn causes xruns when running apps. It looks like other people have found the same problem, but no solutions.
>
>
> I could well be wrong; but my suspicion is not on audio threads - but rather management threads for things like DBus integration, or other "non realtime" operations. If it was the audio thread that was causing the deadlock and spinning, you would hear *no* audio at all.
>
> So next steps I'd suggest running without any extra features (disable Dbus support, or Pulseaudio integration, or ALSA loopback devices, or anything else that's not "core" JACK). See if it still happens..
>  
> Good luck! -Harry
>
> --
>
> http://www.openavproductions.com
>
>
> _______________________________________________
> Jack-Devel mailing list
> [hidden email]
> http://lists.jackaudio.org/listinfo.cgi/jack-devel-jackaudio.org

_______________________________________________
Jack-Devel mailing list
[hidden email]
http://lists.jackaudio.org/listinfo.cgi/jack-devel-jackaudio.org
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: jackdbus process eats two CPU cores for hours on end

Thomas Howe
https://www.youtube.com/watch?v=Aypje2qZPxs - here's a YouTube version of the original video, maybe that will work better.

It does seem to be consistently 6 hours and 13 minutes before the process spike, even when the sample rate is changed from 96 kHz to 192 kHz.

I think the spike is happening in the realtime threads (see https://drive.google.com/open?id=0B2eROo5JatB4am00R2NsS3R6bVk for a low-contrast screenshot of top), so I'm going to try running Jack with realtime disabled this time.

On 5 December 2016 at 09:25, Stéphane Letz <[hidden email]> wrote:
You should try to find out which thread eats the CPU.

Stéphane

> Le 5 déc. 2016 à 01:34, Thomas Howe <[hidden email]> a écrit :
>
> It happens without dbus support (although it is still jackdbus installed).
>
> This time it lasted between 6 hours and 6 hours 30 minutes, and the video shows between 6 hours 12 minutes and 6 hours 13 minutes, so I'm guessing it's about the same duration each time before it spikes, but I'd need to verify this to be sure.
>
> I guess I'll try the jack2 package instead. Any ideas on where I should report this bug?
>
> On 4 December 2016 at 20:43, Thomas Howe <[hidden email]> wrote:
> I haven't measure how long it takes to happen, but it tends to be 'after a while'. This is the first time I've realised it stops after several more hours, which was a surprise. I'm trying to measure how long it takes now.
>
> Is there a way to figure out what's getting processed when this happens?
>
> I'm currently running the process with both of QJackCtl's d-bus boxes unticked to see if (and when) it happens again. PulseAudio isn't installed and I didn't have my ALSA bridge running when recording the video, but I won't start it today anyway just in case it would make a difference. No process spike yet...
>
>
> On 4 December 2016 at 11:01, Harry van Haaren <[hidden email]> wrote:
> On Sun, Dec 4, 2016 at 4:26 AM, Thomas Howe <[hidden email]> wrote:
> I'm having a problem with the jackdbus process; after some hours it starts maxing out two of my four CPU cores, which in turn causes xruns when running apps. It looks like other people have found the same problem, but no solutions.
>
>
> I could well be wrong; but my suspicion is not on audio threads - but rather management threads for things like DBus integration, or other "non realtime" operations. If it was the audio thread that was causing the deadlock and spinning, you would hear *no* audio at all.
>
> So next steps I'd suggest running without any extra features (disable Dbus support, or Pulseaudio integration, or ALSA loopback devices, or anything else that's not "core" JACK). See if it still happens..
>
> Good luck! -Harry
>
> --
>
> http://www.openavproductions.com
>
>
> _______________________________________________
> Jack-Devel mailing list
> [hidden email]
> http://lists.jackaudio.org/listinfo.cgi/jack-devel-jackaudio.org



_______________________________________________
Jack-Devel mailing list
[hidden email]
http://lists.jackaudio.org/listinfo.cgi/jack-devel-jackaudio.org
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: jackdbus process eats two CPU cores for hours on end

Markus Seeber
You could run "perf record" during the time jack consumes a lot of
resources. That way we may get samples of where it spends the most time.
You may need to rebuild jack to include debug symbols so function names
can be seen.

This may at least give a hint on where things are going wrong.

On 12/06/2016 08:03 PM, Thomas Howe wrote:

> https://www.youtube.com/watch?v=Aypje2qZPxs - here's a YouTube version of
> the original video, maybe that will work better.
>
> It does seem to be consistently 6 hours and 13 minutes before the process
> spike, even when the sample rate is changed from 96 kHz to 192 kHz.
>
> I think the spike is happening in the realtime threads (see
> https://drive.google.com/open?id=0B2eROo5JatB4am00R2NsS3R6bVk for a
> low-contrast screenshot of top), so I'm going to try running Jack with
> realtime disabled this time.
>
> On 5 December 2016 at 09:25, Stéphane Letz <[hidden email]> wrote:
>
>> You should try to find out which thread eats the CPU.
>>
>> Stéphane

_______________________________________________
Jack-Devel mailing list
[hidden email]
http://lists.jackaudio.org/listinfo.cgi/jack-devel-jackaudio.org
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: jackdbus process eats two CPU cores for hours on end

John Rigg-16
In reply to this post by Thomas Howe
On Tue, Dec 06, 2016 at 07:03:04PM +0000, Thomas Howe wrote:
> It does seem to be consistently 6 hours and 13 minutes before the process
> spike, even when the sample rate is changed from 96 kHz to 192 kHz.

At 192kHz the JACK frame counter (32 bit uint) overflows and rolls over
to zero after approximately 6hr 12min 49.6s. Coincidence?

John
_______________________________________________
Jack-Devel mailing list
[hidden email]
http://lists.jackaudio.org/listinfo.cgi/jack-devel-jackaudio.org
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: jackdbus process eats two CPU cores for hours on end

Josh Junon
I can do you one better - it's exactly 6 hours, 12 minutes and 49.62 seconds.

192,000 * (2^32 - 1) = 22369.62133 (seconds)

That is exactly 6:12:49.62 worth of time.

On Wed, Dec 7, 2016 at 3:18 AM John Rigg <[hidden email]> wrote:
On Tue, Dec 06, 2016 at 07:03:04PM +0000, Thomas Howe wrote:
> It does seem to be consistently 6 hours and 13 minutes before the process
> spike, even when the sample rate is changed from 96 kHz to 192 kHz.

At 192kHz the JACK frame counter (32 bit uint) overflows and rolls over
to zero after approximately 6hr 12min 49.6s. Coincidence?

John
_______________________________________________
Jack-Devel mailing list
[hidden email]
http://lists.jackaudio.org/listinfo.cgi/jack-devel-jackaudio.org
--
PGP signature: 03A0 B7D0 432E 1514
You may send me encrypted email using my key: https://keybase.io/qix/key.asc

_______________________________________________
Jack-Devel mailing list
[hidden email]
http://lists.jackaudio.org/listinfo.cgi/jack-devel-jackaudio.org
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: jackdbus process eats two CPU cores for hours on end

Thomas Howe
In reply to this post by John Rigg-16
I tried running in non-realtime, and it still happens. I'll try to get the debug symbols and run "perf record" in a few days.

At 192kHz that makes sense, but it happens for the same amount of time at 96kHz which is confusing, but I reckon it's safe to say this is a frame counter-related bug :)

Is there a way to reduce the size of the frame counter from 32 bits to something like 20, so I wouldn't have to leave the process running over 6 hours each time I do a test?

On 7 December 2016 at 10:50, John Rigg <[hidden email]> wrote:
On Tue, Dec 06, 2016 at 07:03:04PM +0000, Thomas Howe wrote:
> It does seem to be consistently 6 hours and 13 minutes before the process
> spike, even when the sample rate is changed from 96 kHz to 192 kHz.

At 192kHz the JACK frame counter (32 bit uint) overflows and rolls over
to zero after approximately 6hr 12min 49.6s. Coincidence?

John
_______________________________________________
Jack-Devel mailing list
[hidden email]
http://lists.jackaudio.org/listinfo.cgi/jack-devel-jackaudio.org


_______________________________________________
Jack-Devel mailing list
[hidden email]
http://lists.jackaudio.org/listinfo.cgi/jack-devel-jackaudio.org
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: jackdbus process eats two CPU cores for hours on end

Florian Paul Schmidt-2
On 12/08/2016 01:51 AM, Thomas Howe wrote:

> I tried running in non-realtime, and it still happens. I'll try to get
> the debug symbols and run "perf record" in a few days.
>
> At 192kHz that makes sense, but it happens for the same amount of time
> at 96kHz which is confusing, but I reckon it's safe to say this is a
> frame counter-related bug :)
>
> Is there a way to reduce the size of the frame counter from 32 bits to
> something like 20, so I wouldn't have to leave the process running over
> 6 hours each time I do a test?

You could try initializing it to something close to the max value :)
Maybe even with attaching gdb to it if your executable has debug symbols
(so you don't need to build it yourself).

Flo


--
https://fps.io
_______________________________________________
Jack-Devel mailing list
[hidden email]
http://lists.jackaudio.org/listinfo.cgi/jack-devel-jackaudio.org
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: jackdbus process eats two CPU cores for hours on end

Thomas Howe
On 11 December 2016 at 15:03, Florian Paul Schmidt <[hidden email]> wrote:
You could try initializing it to something close to the max value :) Maybe even with attaching gdb to it if your executable has debug symbols (so you don't need to build it yourself).

Flo

I haven't figured out how to make jack with debug symbols, but I tried the perf record thing (attached) anyway. It looks from that like MIDI is a factor - in the previous tests I had the MIDI driver set to 'raw', but when set to 'none', there is no glitch after 6h 13m! I'm now repeating the test with 'seq'.

Instructions for initialising jack near the max value would be great! I'll look into gdb next time the spike occurs, which might be in 3h 30m...

_______________________________________________
Jack-Devel mailing list
[hidden email]
http://lists.jackaudio.org/listinfo.cgi/jack-devel-jackaudio.org

perfreport_nospike.txt (163K) Download Attachment
perfreport_spike.txt (235K) Download Attachment
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: jackdbus process eats two CPU cores for hours on end

Thomas Howe
Okay, it seems that it only happens when MIDI is set to 'raw' (not 'seq'), and the selected device is my Scarlett 2i4, so I'll just switch MIDI driver. I'll probably do more testing at some point.

Thanks for the guidance, I doubt I would've isolated the problem without it :)

On 12 December 2016 at 05:49, Thomas Howe <[hidden email]> wrote:
On 11 December 2016 at 15:03, Florian Paul Schmidt <[hidden email]> wrote:
You could try initializing it to something close to the max value :) Maybe even with attaching gdb to it if your executable has debug symbols (so you don't need to build it yourself).

Flo

I haven't figured out how to make jack with debug symbols, but I tried the perf record thing (attached) anyway. It looks from that like MIDI is a factor - in the previous tests I had the MIDI driver set to 'raw', but when set to 'none', there is no glitch after 6h 13m! I'm now repeating the test with 'seq'.

Instructions for initialising jack near the max value would be great! I'll look into gdb next time the spike occurs, which might be in 3h 30m...


_______________________________________________
Jack-Devel mailing list
[hidden email]
http://lists.jackaudio.org/listinfo.cgi/jack-devel-jackaudio.org
Loading...