As of 11-11 My bbs is just randomly starting to crash, running linux mint, have run console and as daemon, bbs is crashing with no errors, or warning.. no segfaults.. when running from terminal when it does crash you can still type on the CLI but no text is echoed, but the command still goes through..

other than that everything is running fine on the BBS, I built a clean build on 11th and 12th and it is still doing it. I bult with options USE_DOSEMU and RELEASE, I Symlink from my exec dir to the gcc.linux.x64.*.release dirs

Until I can find a remedy to the situation I will have to make watchdog script that greps the pid and restarts the BBS if it finds it has crashed.

I wish I could give more info, but it is giving no errors and is not crashing at any certain area or interval.

I think I may have to get GDB involved, it's been a long time since I have done this, it may take a while.. I still have'nt made the watchdog script yet, but pretty sure I can make it without much issue..

I'm running SBBS under a username on my system, all files belong to that user and group, I built sbbs sudo and run sbbs sudo and have it change to UG after binding. I've tried running it every possible way, even as root with all correct permissions, I've even run in init.d. I'm still getting random crashes to the tune of about 7 or 8 a day for the last 2 days. It started with a system that had been stable for at least a month, the only change I have made is installing Solar Realms Elite.. after the crashes started I decided to do a clean build and it's still crashing.

I will try to dig up all the information I can in the next couple days, but will be fairly busy too.

--

Tim Smith (KK4QBN)
KK4QBN BBS

---
* Synchronet * KK4QBN BBS - kk4qbn.com - kk4qbn.synchro.net - Chatsworth GA USA
* Origin: Vertrauen - [vert/cvs/bbs].synchro.net (1:103/705)

Re: random crashes
By: KK4QBN to All on Sun Nov 12 2017 22:41:38

Ok, more on this issue..

Just noticed the crash again, this is the second time I noticed it crash with the SMTP server active and receiving mail from a google server, so for shits and giggles I've disabled the smtp/pop3 lets see how long it stays up now..

--

Tim Smith (KK4QBN)
KK4QBN BBS

---
* Synchronet * KK4QBN BBS - kk4qbn.com - kk4qbn.synchro.net - Chatsworth GA USA
* Origin: Vertrauen - [vert/cvs/bbs].synchro.net (1:103/705)

Re: random crashes
By: KK4QBN to All on Sun Nov 12 2017 10:41 pm

As of 11-11 My bbs is just randomly starting to crash, running linux mint, have run console and as daemon, bbs is crashing with no errors, or warning.. no segfaults.. when running from terminal when it does crash you can still type on the CLI but no text is echoed, but the command still goes through..

What does that mean?

other than that everything is running fine on the BBS, I built a clean build on 11th and 12th and it is still doing it. I bult with options USE_DOSEMU and RELEASE, I Symlink from my exec dir to the gcc.linux.x64.*.release dirs

If you're having troubles, I recommend running a debug build (not release) and you could run it under "gdb" to determine why/how/when the process terminates unexpectedly.

Until I can find a remedy to the situation I will have to make watchdog script that greps the pid and restarts the BBS if it finds it has crashed.

That's not really a fix. See this for more details: http://wiki.synchro.net/howto:gdb

digital man

Synchronet "Real Fact" #84:
The Electronic Frontier Foundation used to run Synchronet (circa 1993).
Norco, CA WX: 59.0oF, 87.0% humidity, 0 mph S wind, 0.00 inches rain/24hrs
--- SBBSecho 3.03-Win32
* Origin: Vertrauen - [vert/cvs/bbs].synchro.net (1:103/705)

Re: random crashes
By: KK4QBN to All on Sun Nov 12 2017 11:05 pm

Re: random crashes
By: KK4QBN to All on Sun Nov 12 2017 22:41:38

Ok, more on this issue..

Just noticed the crash again, this is the second time I noticed it crash with the SMTP server active and receiving mail from a google server, so for shits and giggles I've disabled the smtp/pop3 lets see how long it stays up now..

What was the last log entry before the "crash"?

digital man

Synchronet "Real Fact" #67:
SEXYZ is as a 32-bit replacement for [F]DSZ, CE-XYZ and other protocol drivers. Norco, CA WX: 59.0oF, 87.0% humidity, 0 mph S wind, 0.00 inches rain/24hrs
--- SBBSecho 3.03-Win32
* Origin: Vertrauen - [vert/cvs/bbs].synchro.net (1:103/705)

Re: random crashes
By: Digital Man to KK4QBN on Sun Nov 12 2017 21:52:50

As of 11-11 My bbs is just randomly starting to crash, running linux
mint, have run console and as daemon, bbs is crashing with no errors,
or warning.. no segfaults.. when running from terminal when it does
crash you can still type on the CLI but no text is echoed, but the
command still goes through..

What does that mean?

when sbbs console is running either on tty or in a terminal window, when it would crash the terminal would still work, the command line would.. but the text typed would not show up, but would still work.. let say after the crash I would type 'sudo ./sbbs' and sbbs would start up again..

either way, thats not an issue anymore, it had something to do with smtp and an email coming from google.. I shut smtp off and it has ran all night with no issue.

If you're having troubles, I recommend running a debug build (not release) and you could run it under "gdb" to determine why/how/when the process terminates unexpectedly.

I do believe I did say I would do that somwhere in that message..

its an email coming from google crashing the BBS, on multiple builds.. it would crash before the email was received, but I'm going to go through logs to see if I can find *anything* on it.. other than that I will not be able to replicate it without turning the smtp server back on and having the BBS crash every hour.. I'll have to wait until I have time to monitor the situation.

--

Tim Smith (KK4QBN)
KK4QBN BBS

---
* Synchronet * KK4QBN BBS - kk4qbn.com - kk4qbn.synchro.net - Chatsworth GA USA
* Origin: Vertrauen - [vert/cvs/bbs].synchro.net (1:103/705)

Re: random crashes
By: Digital Man to KK4QBN on Sun Nov 12 2017 21:53:23

Just noticed the crash again, this is the second time I noticed it
crash with the SMTP server active and receiving mail from a google
server, so for shits and giggles I've disabled the smtp/pop3 lets see
how long it stays up now..

What was the last log entry before the "crash"?

why "crash" it was a CRASH.. when something is working, then stops working..
it crashes :)

Nov 12 09:31:02 kk4qbn synchronet: mail 0055 SMTP Connection accepted on port 25 from: 209.85.214.65 port 40907
Nov 12 09:31:02 kk4qbn synchronet: mail 0055 SMTP Hostname: mail-it0-f65.google.com
Nov 12 09:31:03 kk4qbn kernel: [11236.358313] sbbs/smtp[11072]: segfault at 2 ip 00007fab3e6dadcc sp 00007faae9be4440 error 4 in libc-2.19.so[7fab3e690000+1a1000]

looking at syslog it does appear to be a segfault.. it did'nt show this in the terminal ouput.. it would simply "stop working".. or crash.. or however you want to word it..

here is another.

Nov 12 12:48:24 kk4qbn synchronet: mail 0041 SMTP Connection accepted on port 25 from: 209.85.214.65 port 52574
Nov 12 12:48:24 kk4qbn synchronet: mail 0041 SMTP Hostname: mail-it0-f65.google.com
Nov 12 12:48:24 kk4qbn kernel: [23089.420269] sbbs/smtp[15091]: segfault at 2 ip 00007fc956d18dcc sp 00007fc8f37e7440 error 4 in libc-2.19.so[7fc956cce000+1a1000]


I cannot find any other received emails, other than ones that were rejected from DNSBL or ip.can to see whether it's just this particular instance only that will do it or if ANY valid incoming email will do it.. I just received email no less than 3 or so days ago.. so it's hard to say..

--

Tim Smith (KK4QBN)
KK4QBN BBS

---
* Synchronet * KK4QBN BBS - kk4qbn.com - kk4qbn.synchro.net - Chatsworth GA USA
* Origin: Vertrauen - [vert/cvs/bbs].synchro.net (1:103/705)

Re: random crashes
By: KK4QBN to Digital Man on Mon Nov 13 2017 07:12 am

Re: random crashes
By: Digital Man to KK4QBN on Sun Nov 12 2017 21:53:23

Just noticed the crash again, this is the second time I noticed it
crash with the SMTP server active and receiving mail from a google
server, so for shits and giggles I've disabled the smtp/pop3 lets see
how long it stays up now..

What was the last log entry before the "crash"?

why "crash" it was a CRASH.. when something is working, then stops working.. it crashes :)

Okay. I think of a crash as a segfault, GPF, bus error, trap, assertion, etc. If somethign just "exits", it's not really a crash in my book.

Nov 12 09:31:02 kk4qbn synchronet: mail 0055 SMTP Connection accepted on port 25 from: 209.85.214.65 port 40907
Nov 12 09:31:02 kk4qbn synchronet: mail 0055 SMTP Hostname: mail-it0-f65.google.com
Nov 12 09:31:03 kk4qbn kernel: [11236.358313] sbbs/smtp[11072]: segfault at 2 ip 00007fab3e6dadcc sp 00007faae9be4440 error 4 in libc-2.19.so[7fab3e690000+1a1000]

looking at syslog it does appear to be a segfault..

Good, now we're getting somewhere.

it did'nt show this in
the terminal ouput.. it would simply "stop working".. or crash.. or however you want to word it..

I cannot find any other received emails, other than ones that were rejected from DNSBL or ip.can to see whether it's just this particular instance only that will do it or if ANY valid incoming email will do it.. I just received email no less than 3 or so days ago.. so it's hard to say..

If you could run in gdb and get the backtrace (bt) after that crash, then it would probably take me just a matter of minutes of fix.

digital man

Synchronet/BBS Terminology Definition #47:
SMTP = Simple Message Transfer Protocol
Norco, CA WX: 70.4oF, 51.0% humidity, 5 mph NNE wind, 0.00 inches rain/24hrs --- SBBSecho 3.03-Win32
* Origin: Vertrauen - [vert/cvs/bbs].synchro.net (1:103/705)

Re: random crashes
By: Digital Man to KK4QBN on Mon Nov 13 2017 16:12:29

Okay. I think of a crash as a segfault, GPF, bus error, trap, assertion, etc. If somethign just "exits", it's not really a crash in my book.

Nov 12 09:31:02 kk4qbn synchronet: mail 0055 SMTP Connection accepted
on port 25 from: 209.85.214.65 port 40907
Nov 12 09:31:02 kk4qbn synchronet: mail 0055 SMTP Hostname:
mail-it0-f65.google.com
Nov 12 09:31:03 kk4qbn kernel: [11236.358313] sbbs/smtp[11072]:
segfault at 2 ip 00007fab3e6dadcc sp 00007faae9be4440 error 4 in
libc-2.19.so[7fab3e690000+1a1000]

looking at syslog it does appear to be a segfault..

Good, now we're getting somewhere.

Cool!

If you could run in gdb and get the backtrace (bt) after that crash, then it would probably take me just a matter of minutes of fix.

I'll get to it as soon as possible.. until then I've just disabled any email.. it's strange that this has never come up before.. and it was a stable build that ran for at least a couple months straight with no issues..

But here lately I have changed the DNSBL and other options under the mail server options in sbbs.ini So I may have done something SBBS (or the kernel for that matter) does'nt like.

I'll try to get your GDB output asap.

--

Tim Smith (KK4QBN)
KK4QBN BBS

---
* Synchronet * KK4QBN BBS - kk4qbn.com - kk4qbn.synchro.net - Chatsworth GA USA
* Origin: Vertrauen - [vert/cvs/bbs].synchro.net (1:103/705)

Re: random crashes
By: KK4QBN to Digital Man on Mon Nov 13 2017 07:12 am

Re: random crashes
By: Digital Man to KK4QBN on Sun Nov 12 2017 21:53:23

Just noticed the crash again, this is the second time I noticed it
crash with the SMTP server active and receiving mail from a google
server, so for shits and giggles I've disabled the smtp/pop3 lets see
how long it stays up now..

What was the last log entry before the "crash"?

why "crash" it was a CRASH.. when something is working, then stops working.. it crashes :)

Nov 12 09:31:02 kk4qbn synchronet: mail 0055 SMTP Connection accepted on port 25 from: 209.85.214.65 port 40907
Nov 12 09:31:02 kk4qbn synchronet: mail 0055 SMTP Hostname: mail-it0-f65.google.com
Nov 12 09:31:03 kk4qbn kernel: [11236.358313] sbbs/smtp[11072]: segfault at 2 ip 00007fab3e6dadcc sp 00007faae9be4440 error 4 in libc-2.19.so[7fab3e690000+1a1000]

Did you have DEBUG log level enabled at that time? I would expect to see more log entries before the crash. If you could, plese enable Debug log level output as well as the following [Mail] options in sbbs.ini:
- DEBUG_RX_HEADER
- DEBUG_TX

One in particular of interest would be "SMTP Session ID=x" (for that connection that caused the crash).

You should then be able to search for the files SBBS_SMTP.x.* (where 'x' is from Session ID value). If there was a message being received that caused the crash, it should still be there in your Synchronet "temp" directory (e.g. SBBS_SMTP.x.msg), but the other SBBS_SMTP.x.* files would be of interest too.

digital man

Synchronet "Real Fact" #95:
Synchronet v3.14a was released in December of 2006 (1 year after v3.13a). Norco, CA WX: 77.8oF, 31.0% humidity, 0 mph WNW wind, 0.00 inches rain/24hrs --- SBBSecho 3.03-Win32
* Origin: Vertrauen - [vert/cvs/bbs].synchro.net (1:103/705)

Re: random crashes
By: Digital Man to KK4QBN on Tue Nov 14 2017 13:53:39

[1m[34mRe[0m[34m: [1m[36mrandom crashes

mail-it0-f65.google.com
Nov 12 09:31:03 kk4qbn kernel: [11236.358313] sbbs/smtp[11072]:
segfault at 2 ip 00007fab3e6dadcc sp 00007faae9be4440 error 4 in
libc-2.19.so[7fab3e690000+1a1000]

Did you have DEBUG log level enabled at that time? I would expect to see more log entries before the crash. If you could, plese enable Debug log level output as well as the following [Mail] options in sbbs.ini:
- DEBUG_RX_HEADER
- DEBUG_TX

I had those to enabled, BUT I was runnig release version, with INFORMATIONAL as my log level, because I've had so little issue with it.

I promise to get it back up and going to give you some info on itl


I know it did'nt happen until I start using the DNSBL, I used almost every feature. and everytime that google domain would hit it would crash it..

after my mail and a game run, I'll recompile a debug version and run daemonized to see what log entries I get, I'll also read up on GDB and try to make use of it too if needed..

I'd like to give you s much info as possible.

--

Tim Smith (KK4QBN)
KK4QBN BBS

---
* Synchronet * KK4QBN BBS - kk4qbn.com - kk4qbn.synchro.net - Chatsworth GA USA
* Origin: Vertrauen - [vert/cvs/bbs].synchro.net (1:103/705)

On Mon, 13 Nov 2017 06:59:39 -0400, "KK4QBN" <kk4qbn@VERT/KK4QBN>
wrote:

Re: random crashes
By: Digital Man to KK4QBN on Sun Nov 12 2017 21:52:50

As of 11-11 My bbs is just randomly starting to crash, running linux
mint, have run console and as daemon, bbs is crashing with no errors,
or warning.. no segfaults.. when running from terminal when it does
crash you can still type on the CLI but no text is echoed, but the
command still goes through..

What does that mean?

when sbbs console is running either on tty or in a terminal window, when it >would crash the terminal would still work, the command line would.. but the >text typed would not show up, but would still work.. let say after the crash I >would type 'sudo ./sbbs' and sbbs would start up again..

Did you try typing

stty sane


to get your echo back? Sometimes you have to use CTRLM rather then
enter depending how messed up the tty is.

---
* Synchronet * End Of The Line BBS - endofthelinebbs.com
* Origin: Vertrauen - [vert/cvs/bbs].synchro.net (1:103/705)

On Tue, 14 Nov 2017 22:17:02 -0400, "KK4QBN" <kk4qbn@VERT/KK4QBN>
wrote:

Re: random crashes
By: Digital Man to KK4QBN on Tue Nov 14 2017 13:53:39

[1m[34mRe[0m[34m: [1m[36mrandom crashes

mail-it0-f65.google.com
Nov 12 09:31:03 kk4qbn kernel: [11236.358313] sbbs/smtp[11072]:
segfault at 2 ip 00007fab3e6dadcc sp 00007faae9be4440 error 4 in
libc-2.19.so[7fab3e690000+1a1000]

Did you have DEBUG log level enabled at that time? I would expect to see more log entries before the crash. If you could, plese enable Debug log level output as well as the following [Mail] options in sbbs.ini:
- DEBUG_RX_HEADER
- DEBUG_TX

I had those to enabled, BUT I was runnig release version, with INFORMATIONAL as
my log level, because I've had so little issue with it.

I promise to get it back up and going to give you some info on itl

I know it did'nt happen until I start using the DNSBL, I used almost every >feature. and everytime that google domain would hit it would crash it..

What options do you have in your mail settings? I could maybe set mine
the same and see if I can get same issue. I've not really explored
them all yet.


Options=ALLOW_POP3 | ALLOW_RELAY | ALLOW_SYSOP_ALIASES |
DNSBL_THROTTLE | DNSBL_SPAMHASH | USE_SUBMISSION_PORT |
ALLOW_SYSOP_ALIASES

---
* Synchronet * End Of The Line BBS - endofthelinebbs.com
* Origin: Vertrauen - [vert/cvs/bbs].synchro.net (1:103/705)

Re: Re: random crashes
By: Nelgin to KK4QBN on Wed Nov 15 2017 10:24:59

What options do you have in your mail settings? I could maybe set mine
the same and see if I can get same issue. I've not really explored
them all yet.

trncated to fit screen:

Options=DEBUG_RX_BODY | ALLOW_POP3 | ALLOW_RX_BY_NUMBER | ALLOW_SYSOP_ALIASES | USE_SUBMISSION_PORT | USE_TCP_DNS |
SMTP_AUTH_VIA_IP | RELAY_AUTH_LOGIN | DNSBL_REFUSE | DNSBL_BADUSER |
DNSBL_CHKRECVHDRS | DNSBL-THROTTLE | DNSBL_SPAMHASH

--

Tim Smith (KK4QBN)
KK4QBN BBS

---
* Synchronet * KK4QBN BBS - kk4qbn.com - kk4qbn.synchro.net - Chatsworth GA USA
* Origin: Vertrauen - [vert/cvs/bbs].synchro.net (1:103/705)

On 2017 Nov 15 10:24:58, you wrote to KK4QBN:

Options=ALLOW_POP3 | ALLOW_RELAY | ALLOW_SYSOP_ALIASES |
DNSBL_THROTTLE | DNSBL_SPAMHASH | USE_SUBMISSION_PORT | ALLOW_SYSOP_ALIASES

you have "ALLOW_SYSOP_ALIASES" listed twice...

)\/(ark

Always Mount a Scratch Monkey
Do you manage your own servers? If you are not running an IDS/IPS yer doin' it wrong...
... All good things must come to an e
---
* Origin: (1:3634/12.73)

On 2017 Nov 15 13:37:50, you wrote to Nelgin:

What options do you have in your mail settings? I could maybe set mine
the same and see if I can get same issue. I've not really explored
them all yet.

trncated to fit screen:

Options=DEBUG_RX_BODY | ALLOW_POP3 | ALLOW_RX_BY_NUMBER |

remove "ALLOW_RX_BY_NUMBER" and they can't hit you at 1@your.domain.tld

ALLOW_SYSOP_ALIASES | USE_SUBMISSION_PORT | USE_TCP_DNS |
SMTP_AUTH_VIA_IP | RELAY_AUTH_LOGIN | DNSBL_REFUSE | DNSBL_BADUSER |
DNSBL_CHKRECVHDRS | DNSBL-THROTTLE | DNSBL_SPAMHASH

)\/(ark

Always Mount a Scratch Monkey
Do you manage your own servers? If you are not running an IDS/IPS yer doin' it wrong...
... An argument is two people talking and no one listening.
---
* Origin: (1:3634/12.73)