jackd shortcomings in logging and direction for solution

classic Classic list List threaded Threaded
9 messages Options
Reply | Threaded
Open this post in threaded view
|

jackd shortcomings in logging and direction for solution

Federico Grau-2
Hello JACK folk,

I am a bit frustrated with jackd and it's logging abilities.  But, I am
prepared to put some effort to improve the situation.

In my ideal world, jackd would log via syslog with multiple levels of
verbosity.  This would allow for centralized logging and also allow for
differnt level messages to be looked at or ignored (ex: debug vs error
messages).  This logging (at least errors) would happen all the time, so when
there is a problem, it can be inspected and corrected.


As I see things now, jack has either "no logging" (default behaviour) or a
plethora of debug logging (with the "-v --verbose" option passed at the
command line).  While this logging solution may work for a developer doing
debugging, it is not an acceptable solution for logging in production
environments with users that are likely not even aware of jackd.  Even worse,
the current logging does not work at all if jackd is started with the
"JACK_START_SERVER" environment variable.  Even adding adding stdout or stderr
redirections in /etc/jackdrc, the output gets thrown away.


While I am not yet very familiar with the jack codebase, I am willing to
submit a patch that works within the planned jack framework.  I seek feedback
from other veteran jackd developers on how to best proceed with getting better
logging with jackd.

thank you for your feedback,
donfede


--
Federico Grau
Free Software Developer and Sysadmin
Radio Free Asia
2025 M Street, NW
Suite 300
Washington, DC  20036
202-587-2046  Telephone
202-721-7468  Facsimile
CONFIDENTIAL COMMUNICATION
This e-mail message is intended only for the use of the addressee and may
contain information that is privileged and confidential.  Any unauthorized
dissemination, distribution, or copying is strictly prohibited.  If you
receive this transmission in error, please contact [hidden email].


_______________________________________________
Jackit-devel mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/jackit-devel
Reply | Threaded
Open this post in threaded view
|

Re: jackd shortcomings in logging and direction for solution

Paul Davis
On Fri, 2006-06-09 at 14:49 -0400, Federico Grau wrote:
> Hello JACK folk,
>
> I am a bit frustrated with jackd and it's logging abilities.  But, I am
> prepared to put some effort to improve the situation.

i am not saying that i don't think it could be improved. but i think you
misunderstand the overally goal behind jack's design. in my view, if you
need logging to be improved, then jack isn't working as well as it
should and logging won't help fix it.

> As I see things now, jack has either "no logging" (default behaviour) or a
> plethora of debug logging (with the "-v --verbose" option passed at the
> command line).  While this logging solution may work for a developer doing
> debugging, it is not an acceptable solution for logging in production

this isn't true, actually. if jack is built with --enable-debug at
configure time, there is a *lot* more output from -v than in a non-debug
build.

there is already an abstraction built into jackd for "logging", which
writes to a buffer rather than deliver directly to some kind of output.
changing where the output goes is not that hard.

however, i must repeat what i said earlier - if there are problems with
using jack, i don't believe that more or redirected or more controlled
logging is part of the solution.

--p






_______________________________________________
Jackit-devel mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/jackit-devel
Reply | Threaded
Open this post in threaded view
|

Re: jackd shortcomings in logging and direction for solution

Federico Grau-2
On Fri, Jun 09, 2006 at 03:09:01PM -0400, Paul Davis wrote:

> On Fri, 2006-06-09 at 14:49 -0400, Federico Grau wrote:
> > Hello JACK folk,
> >
> > I am a bit frustrated with jackd and it's logging abilities.  But, I am
> > prepared to put some effort to improve the situation.
>
> i am not saying that i don't think it could be improved. but i think you
> misunderstand the overally goal behind jack's design. in my view, if you
> need logging to be improved, then jack isn't working as well as it
> should and logging won't help fix it.
>
> > As I see things now, jack has either "no logging" (default behaviour) or a
> > plethora of debug logging (with the "-v --verbose" option passed at the
> > command line).  While this logging solution may work for a developer doing
> > debugging, it is not an acceptable solution for logging in production
>
> this isn't true, actually. if jack is built with --enable-debug at
> configure time, there is a *lot* more output from -v than in a non-debug
> build.
>
> there is already an abstraction built into jackd for "logging", which
> writes to a buffer rather than deliver directly to some kind of output.
> changing where the output goes is not that hard.
>
> however, i must repeat what i said earlier - if there are problems with
> using jack, i don't believe that more or redirected or more controlled
> logging is part of the solution.
>


Hi Paul and others,

I know jack should be "rock solid", but in our experience "jack is not working
as well as it should be".  While in our environment jack works well 80%+ of
the time, we experience a variety of problems.  Regardless to the source of
the problems, without central logging getting a grasp on what the problems may
be is next to impossible, a bit like flying blind.

There is no doubt that jack is a very useful piece of code (I look forward to
help "jackyfy various Free Software apps over time), but it is definitely
still a work in progress (there are regular cvs commits and new releases
comming out with API changes still happening).  As jack continues to mature, I
cannot imagine a "production" daemon without logging.  This brings back
nightmares of windows 9x and not having any decent logs to analyze when there
is a problem.  Imagine and MTA or scheduler daemon with no logging ...  sure
they are 'supposed' to work correctly all the time ... but there are always
exceptions or times that things need further inspection.  


Thank you for the pointer to the logging abstraction layer, I'll take a look
at that and see if I have any worthwhile contributions.

respectfully,
donfede

--
Federico Grau
Free Software Developer and Sysadmin
Radio Free Asia
2025 M Street, NW
Suite 300
Washington, DC  20036
202-587-2046  Telephone
202-721-7468  Facsimile
CONFIDENTIAL COMMUNICATION
This e-mail message is intended only for the use of the addressee and may
contain information that is privileged and confidential.  Any unauthorized
dissemination, distribution, or copying is strictly prohibited.  If you
receive this transmission in error, please contact [hidden email].


_______________________________________________
Jackit-devel mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/jackit-devel
Reply | Threaded
Open this post in threaded view
|

Re: jackd shortcomings in logging and direction for solution

Paul Davis
On Fri, 2006-06-09 at 15:31 -0400, Federico Grau wrote:

> I know jack should be "rock solid", but in our experience "jack is not working
> as well as it should be".  While in our environment jack works well 80%+ of
> the time, we experience a variety of problems.  Regardless to the source of
> the problems, without central logging getting a grasp on what the problems may
> be is next to impossible, a bit like flying blind.
>
> There is no doubt that jack is a very useful piece of code (I look forward to
> help "jackyfy various Free Software apps over time), but it is definitely
> still a work in progress (there are regular cvs commits and new releases
> comming out with API changes still happening).  As jack continues to mature, I
> cannot imagine a "production" daemon without logging.  This brings back
> nightmares of windows 9x and not having any decent logs to analyze when there
> is a problem.  Imagine and MTA or scheduler daemon with no logging ...  sure
> they are 'supposed' to work correctly all the time ... but there are always
> exceptions or times that things need further inspection.  

in its current incarnation, jackd is not intended (nor is it possible)
to be used by multiple users. therefore, it does not represent a "system
daemon". in fact, i would argue that its not actually a daemon at all.

what kinds of problems are you experiencing and what kind of logging do
you think would help?




_______________________________________________
Jackit-devel mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/jackit-devel
Reply | Threaded
Open this post in threaded view
|

Re: [Rivendell] Re: jackd shortcomings in logging and direction for solution

Federico Grau-2
On Fri, Jun 09, 2006 at 03:36:50PM -0400, Paul Davis wrote:

> On Fri, 2006-06-09 at 15:31 -0400, Federico Grau wrote:
> > I know jack should be "rock solid", but in our experience "jack is not working
> > as well as it should be".  While in our environment jack works well 80%+ of
> > the time, we experience a variety of problems.  Regardless to the source of
> > the problems, without central logging getting a grasp on what the problems may
> > be is next to impossible, a bit like flying blind.
> >
> > There is no doubt that jack is a very useful piece of code (I look forward to
> > help "jackyfy various Free Software apps over time), but it is definitely
> > still a work in progress (there are regular cvs commits and new releases
> > comming out with API changes still happening).  As jack continues to mature, I
> > cannot imagine a "production" daemon without logging.  This brings back
> > nightmares of windows 9x and not having any decent logs to analyze when there
> > is a problem.  Imagine and MTA or scheduler daemon with no logging ...  sure
> > they are 'supposed' to work correctly all the time ... but there are always
> > exceptions or times that things need further inspection.  
>
> in its current incarnation, jackd is not intended (nor is it possible)
> to be used by multiple users. therefore, it does not represent a "system
> daemon". in fact, i would argue that its not actually a daemon at all.
>
> what kinds of problems are you experiencing and what kind of logging do
> you think would help?
>

I am most interested in seeing "why" jackd stops when it does, and to have
that recorded in a central place (syslog is ideal for this, dozens of
workstations can log to a central logging server and allow for easy analysis
if there is a problem).

Other logs that would be interesting are when jack kicks a client (why, when,
how often).  This will allow me to identify a problem, possibly with jack or
possibly with the client.


The most common example of jack failures we are seeing recently is when
starting an app that will start jack (via the JACK_START_SERVER env); the app
dies and jack is no-where to be seen.   Starting the app a second time works
most of the time (but there is obviously a problem present).  The app is
Rivendell's "cae" and jack is 0.100.0 on a debian stable system.

We have had other other times when jack has failed, but I do not have
worthwhile records of that now (thus the logging would be ideal).  

I am also now trying to study how jackd exits (and cleans up), especially when
started within an X session and dying on exit of the X session.  I am unable
to get those logs out of jack when it's started with teh JACK_START_SERVER
env.


As I am working on deploying this project, jack is a key piece of code that
HAS to work.  Before I deploy it to dozens of workstations, the sysadmin in me
says "I have to have eyes" for WHEN there is a problem (there will be one
sooner or later) so I can give a proper answer to my users and take actions to
correct it.

regards,
donfede



--
Federico Grau
Free Software Developer and Sysadmin
Radio Free Asia
2025 M Street, NW
Suite 300
Washington, DC  20036
202-587-2046  Telephone
202-721-7468  Facsimile
CONFIDENTIAL COMMUNICATION
This e-mail message is intended only for the use of the addressee and may
contain information that is privileged and confidential.  Any unauthorized
dissemination, distribution, or copying is strictly prohibited.  If you
receive this transmission in error, please contact [hidden email].


_______________________________________________
Jackit-devel mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/jackit-devel
Reply | Threaded
Open this post in threaded view
|

Re: [Rivendell] Re: jackd shortcomings in logging and direction for solution

Jack O'Quin
Thanks for the explanation.  I personally have no objection to adding
a syslog option to jackd.  The main difficulty would probably be doing
a good job of classifying the levels of all those messages.  That would
be worthwhile, but a lot of work.

I believe that quite a few --verbose messages from jackd are actually
errror messages that should appear even without that option.  Fixing
some of those would probably help a lot of new users figure out what
has gone wrong, even without syslog.
--
 joq


_______________________________________________
Jackit-devel mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/jackit-devel
Reply | Threaded
Open this post in threaded view
|

Re: [Rivendell] Re: jackd shortcomings in logging and direction for solution

John Rigg-3
In reply to this post by Federico Grau-2
On Fri, Jun 09, 2006 at 03:58:31PM -0400, Federico Grau wrote:
> The most common example of jack failures we are seeing recently is when
> starting an app that will start jack (via the JACK_START_SERVER env); the app
> dies and jack is no-where to be seen.   Starting the app a second time works
> most of the time (but there is obviously a problem present).  The app is
> Rivendell's "cae" and jack is 0.100.0 on a debian stable system.

You are aware that Debian stable uses libpthread-0.60 in libc6? This is
broken so threaded apps won't work properly. You need to tell jackd to
use an earlier thread model by using LD_ASSUME_KERNEL, eg.:
LD_ASSUME_KERNEL=2.4.22 jackd -R -d alsa ...
Clients that are supposed to start jackd will have to be configured to
do this.

If you're using AMD64 you won't have this option as no earlier thread model
is provided in the kernel. Debian 3.1r2 has some thread related changes
in libc6 for AMD64 which might fix it, but I don't think it's out yet.

John


_______________________________________________
Jackit-devel mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/jackit-devel
Reply | Threaded
Open this post in threaded view
|

Re: [Rivendell] Re: jackd shortcomings in logging and direction for solution

John Rigg-3
On Sat, Jun 10, 2006 at 09:44:29AM +0100, John Rigg wrote:
> If you're using AMD64 you won't have this option as no earlier thread model
> is provided in the kernel. Debian 3.1r2 has some thread related changes
> in libc6 for AMD64 which might fix it, but I don't think it's out yet.

Hmm. I could be wrong about earlier thread models not being in the AMD64
kernel. I do remember seeing somewhere that only NPTL works on AMD64 but
I can't remember where. Perhaps LD_ASSUME_KERNEL works on AMD64 after
all. I switched to Debian testing/unstable some time ago to avoid all
this.

John


_______________________________________________
Jackit-devel mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/jackit-devel
Reply | Threaded
Open this post in threaded view
|

Re: [Rivendell] Re: jackd shortcomings in logging and direction for solution

John Rigg-3
In reply to this post by John Rigg-3
On Sat, Jun 10, 2006 at 09:44:29AM +0100, John Rigg wrote:

> On Fri, Jun 09, 2006 at 03:58:31PM -0400, Federico Grau wrote:
> > The most common example of jack failures we are seeing recently is when
> > starting an app that will start jack (via the JACK_START_SERVER env); the app
> > dies and jack is no-where to be seen.   Starting the app a second time works
> > most of the time (but there is obviously a problem present).  The app is
> > Rivendell's "cae" and jack is 0.100.0 on a debian stable system.
>
> You are aware that Debian stable uses libpthread-0.60 in libc6? This is
> broken so threaded apps won't work properly. You need to tell jackd to
> use an earlier thread model by using LD_ASSUME_KERNEL, eg.:
> LD_ASSUME_KERNEL=2.4.22 jackd -R -d alsa ...
> Clients that are supposed to start jackd will have to be configured to
> do this.

Looking into this further I see that jackd 0.99 (the version in Debian
stable) was patched to work around the pthreads/NPTL problem. You
might still need to use LD_ASSUME_KERNEL with jack clients, though.

John


_______________________________________________
Jackit-devel mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/jackit-devel