mod_perl debugging

rrmesser wrote on Mon Mar 24 07:40:22 CET 2008:

I am trying to get Eclipse to work with mod_perl for debugging.  I'm using
Apache/2.2.8, mod_perl/2.0.4, with Perl v5.10.

Anyway, it seems tantalizingly close to working -- when I make a request
from the mod_perl Apache, Eclipse brings up the handler and lets me step
through code.  But there are some odd things that prevent me from getting
it to work completely, such as:

- when I use the stock perl5db.pl (that comes with Apache::DB), and then
resume, I never hit another break point.  The next request just continues,
seemingly ignoring the debugger -- like Eclipse cleared all breakpoints
and never send them again, or maybe it is always sending a "c" for continue
with every request?

- If I use the EPIC version of perl5db.pl (which comes with epic_breakpoints.pm
in the debug package), then I never stop at any breakpoints at all -- the
mod_perl apache continues along and it looks as if the debugging isn't running,
but rather just the mod_perl apache in regular mode.  But I know the debugger
is running, it gives the normally start message and I can see its warn statements
and such.

- Eclipse also continues to show "waiting for debugger connection" after
I start the debug session.  And eclipse starts to listen on ports 5001 and
5002 for something, I'm not sure what.  I can telnet to those ports but
they don't seem to be doing much, and I can't find any documentation for
what eclipse is expecting there.

Any assistance is appreciated.  I've been using the mod_perl debugger for
four or five years now, so I am experienced with the command line version,
and also have used it under ddd.  But I'd much rather get it working under
Eclipse if possible.  I'm willing to work with somebody experienced with
EPIC to make it happen, and write up documentation after it is working,
since I think it would generally be a useful feature for others working
with Eclipse and Perl.  Thanks,

Rob

sureei666 wrote on Mon Mar 24 08:43:25 CET 2008:

Hi Robert,
         Am beginner of using EPIC. I have an issue with array while debugging.
My script specifies $[ = 1; as base value of array is 1. But still debugger
treats arrays from index 0 instead of 1. Hope you might be having good idea
to overcome this problem. If possible please help me out from this weird
issue.

jploski wrote on Mon Mar 24 13:58:36 CET 2008:

Using an unmodified version of perl5db.pl from Apache would disable EPIC's
insertion of breakpoints into newly loaded source files, possibly more.
Bad idea.

Does Apache's perl5db.pl differ from the stock one shipped with Perl? If
so, that might be a reason why substituting EPIC's version breaks something.

EPIC is waiting on redirected stderr/stdout on the ports 5001, 5002. You
can turn off such redirection by setting an option in the launch configuration.

There have been some past threads about getting EPIC to work with mod_perl
(I don't remember whether any resolution has been reached). If you'd like
to have a closer look at the code, read http://e-p-i-c.sf.net/devguide/devguide.html
Look at org.epic.debug.db.ResumeCommand to see how it decides whether to
stop or continue after detecting that the interpreter was suspended.

rrmesser wrote on Mon Mar 24 19:37:01 CET 2008:

Jan,

The perl5db.pl file that ships with Apache::DB does differ slightly (one
line) from the version that comes with Perl itself.  During the installation
of Apache::DB, it copies over the stock perl5db.pl and applies a patch.
 (Changes "END" to "sub db_END", here's the link if it helps: http://search.cpan.org/src/FWILES/Apache-DB-0.13/Makefile.PL)
 But I applied that patch to EPIC's perl5db.pl, and still it doesn't recognize
breakpoints.

I knew the regular perl5db.pl probably wouldn't work, but I tried it as
a test.  It does in fact break at the start of the program, which is the
default for the command line mod_perl debugging interface.  So that led
me to believe debugging mod_perl through EPIC could work.

There have been a few threads over the past few years about mod_perl and
EPIC, but none of them indicate that somebody actually got it to work completely.
 I emailed the author of the most recent thread, and if I hear back from
him I'll post it here.

In the meantime I'll look at the devguide link you sent, and also try to
figure out epic_breakpoints.pm.  Somehow it seems the breakpoints aren't
getting sent successfully from EPIC to the mod_perl debugging process, so
it just runs with no pauses.  The "remote debugger" console window in Eclipse
also shows nothing, so it's hard to tell exactly what's going wrong.

As far as the ports 5001/5002 -- where is the option turn off waiting for
that redirection?  The debugger launch configuration has a place to enter
the port for listening to the remote debugger,  and there are "capture output"
and "create debug package" options, but I don't see anything about listening
on other ports.  Thanks,

Rob

jploski wrote on Mon Mar 24 19:50:05 CET 2008:

Re ports: the "capture output" option is what is meant.

Another debugging tip is to enable the "Debug console" in EPIC Preferences.
This allows you to watch the communication of EPIC with the perl5db.pl process
(see past threads for details). This is probably what you expected to see
in the "remote debugger" console window.

rrmesser wrote on Mon Mar 24 23:24:02 CET 2008:

Jan,

Thanks, enabling the "Debug console" did help quite a bit.  I was able to
see that there was an issue with finding the path to some of the files in
my project.  Once I corrected that, the breakpoints worked.

However, even with "capture output" unchecked, it seems that EPIC still
starts listening on ports 5001 and 5002 and expecting to get something there
(I know because I can telnet to those ports after I start the debug session
but not before), and it displays a message "Waiting for debugger connection...".
 In fact, if I run the debug process in the foreground, a modal dialog box
with that message appears.  When I send a hit to the mod_perl apache, it
does bring up the correct file in Eclipse, but I can't actually debug because
the dialog is in the way, and so I can't click "resume" or other buttons.

If I start the debugger with "Launch in background" enabled, then things
work better -- the dialog isn't in the way anymore, so once my code hits
a breakpoint, it stops, and then I can step with F6 or click resume to continue.
 All good, but I can still see in the "Progress" tab the "waiting for debugger
connection" message.  Then exactly 30 seconds after I started the debug
process in Eclipse, EPIC sends a 'q' to the debugger process (which it has
connected to, running on port 5000 which I specified in the launch config
and in PERLDB_OPTS).  That "q" ends the debugger session, and I have to
restart the debugging process and get EPIC to reconnect.  Because it's exactly
30 seconds, and also because I am seeing some log messages like "Timed out
while waiting for IO redirect from the debugged process," I'm thinking EPIC
is looking for a connection on either port 5001 or 5002, and when it doesn't
get it, it sends a "q" to the debug process on port 5000.

Anyway, I do have it working on some level, since I can debug with mod_perl
for 30 seconds at a time.

To make it useful, as a hack I think I could go in and find that 30 second
timeout value and change it to a very large number.  It would be better
of course to make EPIC realize that it only needs to connect on the main
user-specific port specified in the Perl Remote configuration.  After all
with mod_perl, stderr (warn statements) normally go into the apache logs
as well as to the debug console, and stdout goes back to the web browser.

Perhaps the "capture output" option needs some adjustment?  Or there should
be some other option to facilitate debugging under mod_perl?

Rob

jploski wrote on Tue Mar 25 22:52:56 CET 2008:

I believe I could reproduce the problem with a remote debug configuration
with "capture output" disabled (without mod_perl). I committed a fix to
be released in 0.6.22. Unfortunately, the CVS access from the shell server
at sf.net is currently not working, so the release is going to be delayed
(I just hope they didn't disable it for good for some reason). You can grab
the fix from CVS if you're impatient.

rrmesser wrote on Wed Mar 26 01:36:13 CET 2008:

Sounds good, thanks.  I'll wait for the 0.6.22 release and then test it
and reply back again with the results.  Also after that I'd be happy to
write up a "how to debug under mod_perl with EPIC" page if that would be
helpful.

jploski wrote on Wed Mar 26 20:20:46 CET 2008:

0.6.22 is out. A mod_perl debugging tutorial would be surely appreciated.

rrmesser wrote on Thu Mar 27 18:23:53 CET 2008:

Ok, I tried it out and that changed fixed the timeout issue, thanks much.
 So now it works great as far as controlling the breakpoints, stepping and
continuing and so on.

However, now there is one more thing before I can really use it.  The variable
view always reports "an error occurred while retrieving variables from the
debugger process."  I looked at old forum posts and saw this one about "missing
right curly...": https://sourceforge.net/forum/forum.php?thread_id=1528954&forum_id=258688.

But that doesn't seem to be my issue.  I'm using perl 5.10, which already
has a fix in perl5db.pl which is similar to that patch.

So I added some print statements to send perl5db.pl stuff to logs (debugging
the old fashioned way), and it looks like sometimes the variable dump code
is getting sent in chunks that are broken up.  For example, sometimes this
comes across as one transmission, which works fine:

---------------------
($@, $!, $^E, $,, $/, $\, $^W) = @saved;package IS::Form; $^D = $^D | $DB::db_stop;
;{    
do 'dumpvar_epic.pm' unless defined &dumpvar_epic::dump_lexical_vars;
    
my $offset = 0;
my $varexpr = <<'EOT';
$h->{'@qerrlist'}
EOT
my $subref = \&dumpvar_epic::dump_array_expr;
my $savout = select($DB::OUT);
my $savbuf = $|;
$| = 0;
$subref->($offset, $varexpr);
$| = $savbuf;
print "";
select($savout);
};;
---------------------

But then the next time, it'll get split, so there are two separate sends,
like this:

-------------------------------------------------------------------------------
;{    
do 'dumpvar_epic.pm' unless defined &dumpvar_epic::dump_lexical_vars;
    
my $offset = 0;
my $varexpr = <<'EOT';
$h->{'@qerrlist'}
EOT
my $subref = \&dumpvar_epic::dump_array_expr;
my $savout 
--------------------------------------------------------------------------------
--------------------------------------------------------------------------------
 = select($DB::OUT);
my $savbuf = $|;
$| = 0;
$subref->($offset, $varexpr);
$| = $savbuf;
print "";
select($savout);
};
---------------------------------------------------------------------------------

perl5db.pl tries to eval the first chunk and chokes on it.  Any ideas? 
To me it seems like either some kind of transmission error, or EPIC is breaking
up the send somehow?

jploski wrote on Thu Mar 27 19:21:32 CET 2008:

It would help if you could tell whether there are two sends + two receives
or one send + two receives causing this problem (you can probably find out
using strace).

rrmesser wrote on Thu Mar 27 23:35:49 CET 2008:

Ok, never mind about the issue with the variable view not working.  I read
a few more posts on the forum about people having trouble with the default
java version, and so I upgraded to java 1.6 and the problem went away. 
I am using a CentOS 5.1 VM, and the default java there is the gcj version
1.4.2.  Apparently that version somehow split up messages to the debugging
port, but that's not an issue now that I upgraded.

So now it's all good, and I'll write up some brief instructions and post
it to the forum in the next few days.

The only remaining issue is relatively minor because I can work around it.
 But here it is anyway -- when the debugger isn't active, the whole Eclipse
window is frozen.  There can be no input or output, and resizing the window
or moving panes around doesn't work.

To explain a bit more -- the typical cycle of debugging under mod_perl goes
something like this:

1) start the debugging process in EPIC, so it listens for debugging output
on the port you've chosen
2) start the apache mod_perl debugger from the command line
3) hit a web page in your app
4) the debugger becomes active.  step through the code, set breakpoints,
look at variables, etc.
5) when done, hit continue (resume).  The page is sent to the browser which
displays it.
6) hit another page in your browser, or refresh to hit the same page again.
then go back to step 4 until you're done.

Eclipse is frozen between steps 5 and 6.  So you can't edit files until
you go to another page to make the debugger active.  In particular, let's
say you want to temporarily browse to the right place in your webapp before
setting another break point.  So you disable all breakpoints for a minute.
 But now you can never add breakpoints back, because Eclipse is frozen the
whole time.

The workaround is to terminate the apache mod_perl debugging process, at
which point Eclipse wakes up.  You can then restart from step 1 and go from
there.  So it's not a huge deal, but it would be nice to fix.

In any event, thanks for your help in getting EPIC working for me in debugging
under mod_perl.

jploski wrote on Sun Mar 30 21:01:11 MEST 2008:

I got mod_perl working on my development machine today. I'm going to look
into the GUI-thread-lockup-after-resume problem some time next week...

jploski wrote on Tue Apr  1 19:56:08 MEST 2008:

The good news is that I now got the mod_perl debugger working with EPIC,
the very same sequence of steps you describe. The bad news is that I cannot
reproduce the GUI thread lockup you experience. When I hit resume, the GUI
remains responsive just as one expects it should.

I do have GUI thread lockup problems, but only when terminating the debugger
(with the Terminate button). This is because the debugger does not respond
to 'q' and does not even respond to SIGTERM. I have to kill -9 the httpd
-X process and after that the session in EPIC also terminates.

I performed my tests on the sample "mod_perl rocks" handler from the documentation,
but I see there's also this ModPerl::Registry wrapper for legacy scripts,
which might behave differently (though I doubt it). In any case, let me
know which one you use for testing.

By the way, I see no difference in behavior with perl5db.pl from EPIC or
with perl5db.pl from EPIC with Apache::DB diffs. I discovered and fixed
some minor problems in epic_breakpoints.pm (complaining about undefined
references) as well as in exception handling (port 5000 was not being freed
properly), but these were not related to the GUI lockups you reported.

rrmesser wrote on Tue Apr  1 23:45:03 MEST 2008:

Ok, glad you were able to get the debugger working under mod_perl, even
though you couldn't reproduce my problem.  What version of eclipse are you
using by the way?  I was using 3.2, but then when I saw your message that
you didn't have the same problem I did, I upgraded to 3.3.2, re-installed
EPIC and other plug-ins, and tried again.

Now the behavior with 3.3.2 is different for me, but not consistent.  It
seems like after the first few web requests, when I resume, the GUI is locked
up.  Then later it isn't.  I haven't been able to nail down exactly when
it locks up to give you more information.

I can say that regardless, if I ever remove all breakpoints I cannot debug
again without terminating the process.  If I put breakpoints back in (when
the GUI is not locked), they are never sent to the debugger, because EPIC
seems to want to send them immediately.  But the debugger is basically not
accepting input while it is waiting for the next request.  That's the same
as with mod_perl debugging with a simple console window.  For EPIC to send
those breakpoints, it would have to wait until the next web request, then
send them at the start of that, before the first 'c' I'd think.  Or maybe
you have some other way to do it?

As far as the handler -- I am using a custom handler, not ModPerl::Registry.
 But like you say, that shouldn't make any difference.

I do also have to kill the debug process.  It was never kill -9 until perl
5.10 though, because there is an issue with Apache::DB and 5.10.

Sometimes after killing, when I restart I get "Cannot listen on local port
debug port 5000.  Check that it is not bound by another application."  So
I have to shutdown/restart eclipse.  It sounds like you fixed that issue.
 Is that in cvs?

One final question -- I started writing up a brief how-to on mod_perl debugging
with Eclipse, but wasn't sure how best to post it.  Maybe just a new thread
in the forum?

jploski wrote on Wed Apr  2 00:28:34 MEST 2008:

I was using 3.2 for my testing, I'll check with 3.3 later.

About putting breakpoints: I can't see how the issue would be different
in normal Perl (say, with a long loop) and mod_perl. In both cases you can
manipulate breakpoints in Eclipse when the debugger is not ready, and EPIC
should be buffering the breakpoint commands until the next suspend (I'll
check whether it works).

I have to kill -9 even though I'm using perl 5.8.8 (with Apache 2.2.8 and
mod_perl 2.0.3).

The "cannot listen" issue should be indeed fixed in CVS.

I think it will be best to include the mod_perl how-to in the EPIC User
Guide (somewhere near the description of the "Perl Remote" launch configuration
type). So it doesn't really matter how you post it. You could edit guide.xml
in the org.epic.doc project directly if you are familiar with DocBook.

jploski wrote on Wed Apr  2 17:16:00 MEST 2008:

I have the same behavior with 3.3 and 3.2 - no GUI lockups.

I found adding/removing breakpoints while the debugger was not suspended
to be unreliable. Fixed it in CVS.