A long-requested feature of SBBSecho is automatic addition of "bad areas" (messages received for an area not present in the SBBSecho Area File, e.g. areas.bbs) and the re-tossiong of previously received "bad messages" into the newly added sub-boards. This feature would allow a sysop to just turn on the firehose from their hub (e.g. AreaFix "+ALL") and then have the areas created on demand (upon incomming traffic), automatically without sysop intervention
of any kind.

SBBSecho has long had the ability to store received "bad messages" into a
"bad message" area (sub-board). The original "AREA" control line is retained
in each message header, so it is theoretically possible for something to re-toss these "bad messages" into the correct corresponding local sub-boards later (e.g. after the areas were created and added to the area file).

The next required step is to keep track of the "bad areas". I added this in SBBSecho v3.01 back in June: It'll maintain a file (by default, named data/badareas.lst) which keeps track of the "bad areas", in echolist format (with corresponding area descriptions, if you have "Additional EchoLists" configured in sbbsecho.ini/echocfg). As areas are added to your configuration and Area File, they are automatically removed from your badareas.lst by SBBSecho. If you're running 3.01, take a look at this file, it might be insightful.

The third step is to allow this "bad areas" list to be added to the sysop's configuration automatically. Working towards this goal, I've added a new
SCFG command-line option: -import=<filename> which can be used to import
an area list (e.g. badareas.lst) into a message group (use -g<group name>
to specify the target group). This command doesn't display any dialogs
or require any input; it just imports the area list (or tries to) and
exits back to the command prompt (displays errors and returns non-zero errorlevel on failure). Now this is not how a sysop historically uses SCFG,
but it seemed like a nice feature to have for quickly and possibly automatically importing a message area list (subs.txt, areas.bbs, and QWK control.dat formats are also supported). This was a pretty large change to
make in SCFG and I'd appreciate some test results from sysops.

The fourth step would be the automatic addition of the newly-created
sub-boards to the SBBSecho Area File (e.g. areas.bbs). I've not done anything in this regards yet.

The fifth step, if the sysop wanted, would be to re-toss from the "bad messages" sub-board into the appropriate local sub-boards and delete the messages from the "bad messages" area. This is likely a feature I will implement in SBBSecho soon, in the form of a command-line option since
I don't think you'd want to be constantly trying to re-toss the bad area
for every invocation of SBBSecho. I don't think this feature will be tough.

Now, in looking at my badareas.lst, I see a lot of echoes I don't think *I* would want auto-added back into my configuration. I recall removing many
of these echoes (areas) because they contained nothing but duplicate echo
rules or moderator notices and they received no "real" traffic. So I was thinking about adding statistics to the badareas.lst and then decided against it:
1. It has value being in the EchoList format (same as BACKBONE.NA/FIDONET.NA) 2. It could be useful to collect statistics for ALL areas, not just the bad

Collected echostats could be used to filter the badareas.lst file before importing (creating new areas) for example. You could say (theoretically),
only import the areas that receive 10 unique messages in the past month or something like that. Then an area would have to be listed in your badareas.lst for at least a month and it would have to receive 10 unique (non-dupe)
messages in one month (30 day?) period before it would be auto-added.
Something like that (I'm just spitballing). Of course each sysop would be
free to set their own filtering criteria, or not filter at all and just auto-add every "bad area" as they popped up.

And then the areas you do have in your configuration could possibly be automatically removed based on similar statistics/filtering rules (e.g. remove areas that don't receive more than one message a month on average).

So I've begun work on a feature I'm calling "echostats", whereby SBBSecho
keeps track of messages received, imported, exported, dupes, and circular
for every echo area (good and bad). A sysop would be free to use this information or just ignore it (or even disable the feature entirely in SBBSecho). I'm using a .ini file format, so it's not something that would be easy to import into a spreadsheeteet, but it would be trivial to write utilities in JS or whatever to use the data. Most of the data could be
obtained by parsing the sbbsecho.log file, but I'm collecting more detail
about the messages (first and last, matching each criteria) and collating them in one easily digestable file.

Here's an example from just one echo (OTHERNETS):

[OTHERNETS]
FirstReceived.to = All
FirstReceived.from = Gert Andersen
FirstReceived.subj = Join Linuxnet
FirstReceived.length = 0
FirstReceived.msgtime = Oct 24 2017 09:45:22
FirstReceived.localtime = Oct 24 2017 02:14:40
FirstReceived.origaddr = 2:280/464
FirstReceived.pkt_orig = 2:280/464
LastReceived.to = All
LastReceived.from = Bill McGarrity
LastReceived.subj = SportNet Application!
LastReceived.length = 0
LastReceived.msgtime = Oct 25 2017 00:01:23
LastReceived.localtime = Oct 25 2017 15:33:57
LastReceived.origaddr = 2:280/464
LastReceived.pkt_orig = 2:280/464
TotalReceived = 1
FirstImported.to = All
FirstImported.from = Gert Andersen
FirstImported.subj = Join Linuxnet
FirstImported.length = 0
FirstImported.msgtime = Oct 24 2017 09:45:22
FirstImported.localtime = Oct 24 2017 02:14:50
FirstImported.origaddr = 2:230/150
FirstImported.pkt_orig = 2:280/464
LastImported.to = All
LastImported.from = Bill McGarrity
LastImported.subj = SportNet Application!
LastImported.msg_id = 3617.2othernet@1:266/404 1e5550b4
LastImported.pid = SMBUTIL 2.33-Win32 r1.113 Oct 13 2017 MSC 1800 LastImported.tid = SBBSecho 3.01-Win32 r3.39 Oct 13 2017 MSC 1800 LastImported.length = 3095
LastImported.msgtime = Oct 25 2017 00:01:23
LastImported.localtime = Oct 25 2017 15:34:05
LastImported.origaddr = 1:266/404
LastImported.pkt_orig = 2:280/464
TotalImported = 1
TotalExported = 0
FirstDuplicate.to = All
FirstDuplicate.from = Gert Andersen
FirstDuplicate.subj = Join 1 AdvnetureNet
FirstDuplicate.length = 0
FirstDuplicate.msgtime = Oct 24 2017 09:45:24
FirstDuplicate.localtime = Oct 24 2017 03:41:00
FirstDuplicate.origaddr = 2:230/150
FirstDuplicate.pkt_orig = 1:218/700
LastDuplicate.to = All
LastDuplicate.from = Gert Andersen
LastDuplicate.subj = Join 1 AdvnetureNet
LastDuplicate.length = 0
LastDuplicate.msgtime = Oct 24 2017 09:45:24
LastDuplicate.localtime = Oct 24 2017 03:41:00
LastDuplicate.origaddr = 2:230/150
LastDuplicate.pkt_orig = 1:218/700
TotalDuplicate = 0
TotalCircular = 0

There is a bug with the "Total" values (they should be higher), so ignore those. And I'm working on capturing the message-IDs and origin address from
all incoming messages before being imported, so those details will be
available for more "types" of messages. But you get the idea.

So if you have ideas about what kinds of statistical data you mind want to use (e.g. for auto-adding or auto-removing of areas) or for any other reason, let know. Or any other thoughts on this long-time still in-progress feature/goal.

digital man

Synchronet "Real Fact" #58:
The last version of Synchronet to run on MS-DOS and OS/2 was v2.30c (1999). Norco, CA WX: 87.1oF, 15.0% humidity, 2 mph ESE wind, 0.00 inches rain/24hrs --- SBBSecho 3.01-Win32
* Origin: Vertrauen - [vert/cvs/bbs].synchro.net (1:103/705)

Digital Man wrote to All <=-

@VIA: VERT
@TZ: c1e0
A long-requested feature of SBBSecho is automatic addition of "bad
areas" (messages received for an area not present in the SBBSecho Area File, e.g. areas.bbs) and the re-tossiong of previously received "bad messages" into the newly added sub-boards. This feature would allow a sysop to just turn on the firehose from their hub (e.g. AreaFix "+ALL") and then have the areas created on demand (upon incomming traffic), automatically without sysop intervention of any kind.

<snip>

Sounds like some really neat features in the pipeline. I'd like to be able to setup rules so echoes from a particular zone:net or a particular uplink went to a specific group, for example, new echoes from my feed in zone 21 would be created in my FSXNET group, and obviously those from my Fido uplink would go into the Fidonet group, etc. Since we're using SCFG anyway, an interactive mode would be nice, so I could choose the order of the newly created echoes. While not really important for FTN nets, it does become more so, if I'm gating to QWK (which I do for a couple of nets).

While I have got the hang of manually setting up new echoes and nets in SCFG and AREAS.BBS, automation would be nice to have.

While on the subject of adding and importing message areas, another noce to have would be to be able to automatically tweak security settings when importing areas in Synchronet's native (SCFG) format, since each board has different levels for different things - for example, if an area has a minimum security level of 30 for normal BBS users on the source, I may use 25 on my BBS, etc. It would be nice to be able to tell SCFG to make those adjustments on importing areas. Yes, I know you can clone, but that can have issues, when there's sysop echoes or others that require unique settings in the mix.


... File not found. Should I fake it? (Y/N)
--- MultiMail/Win32 v0.49
* Synchronet * Freeway BBS in Bendigo, Australia.
* Origin: Vertrauen - [vert/cvs/bbs].synchro.net (1:103/705)

Re: Re: SBBSecho - bad areas and echostats
By: Vk3jed to Digital Man on Fri Oct 27 2017 06:56 am

Sounds like some really neat features in the pipeline. I'd like to be able to setup rules so echoes from a particular zone:net or a particular uplink went to a specific group, for example, new echoes from my feed in zone 21 would be created in my FSXNET group, and obviously those from my Fido uplink would go into the Fidonet group, etc. Since we're using SCFG anyway, an interactive mode would be nice, so I could choose the order of the newly created echoes. While not really important for FTN nets, it does become more so, if I'm gating to QWK (which I do for a couple of nets).

That's something good think about. If you ran multiple instances/configurations of SBBSecho, you could have a separate badareas.lst files for each instance/configuration. Then you'd certainly have control over which groups you were auto-adding areas to.

If you're talking about a single instance/configuration of SBBSecho shared betwen multiple networks, then I'll have to give that some thought about how to do that.

While I have got the hang of manually setting up new echoes and nets in SCFG and AREAS.BBS, automation would be nice to have.

While on the subject of adding and importing message areas, another noce to have would be to be able to automatically tweak security settings when importing areas in Synchronet's native (SCFG) format, since each board has different levels for different things - for example, if an area has a minimum security level of 30 for normal BBS users on the source, I may use 25 on my BBS, etc. It would be nice to be able to tell SCFG to make those adjustments on importing areas. Yes, I know you can clone, but that can have issues, when there's sysop echoes or others that require unique settings in the mix.

The subs.txt file format is not ideal. I'll likely be adding a .ini import/export format in the future and that'd make it easy(er) to remove or filter out the options you don't want to import or over-ride your defaults/template. For now, just editing the subs.txt first might be your best option.

digital man

This Is Spinal Tap quote #20:
Well, I'm sure I'd feel much worse if I weren't under such heavy sedation. Norco, CA WX: 85.8oF, 28.0% humidity, 6 mph ENE wind, 0.00 inches rain/24hrs --- SBBSecho 3.01-Win32
* Origin: Vertrauen - [vert/cvs/bbs].synchro.net (1:103/705)

Digital Man wrote to Vk3jed <=-

That's something good think about. If you ran multiple instances/configurations of SBBSecho, you could have a separate badareas.lst files for each instance/configuration. Then you'd
certainly have control over which groups you were auto-adding areas to.

If you're talking about a single instance/configuration of SBBSecho
shared betwen multiple networks, then I'll have to give that some
thought about how to do that.

Well, I do run a single instance, so we're in the latter scenario.

The subs.txt file format is not ideal. I'll likely be adding a .ini import/export format in the future and that'd make it easy(er) to
remove or filter out the options you don't want to import or over-ride your defaults/template. For now, just editing the subs.txt first might
be your best option.

Better, but still rather tedious, when there's a couple dozen areas to import. :)


... Advertising is the rattling of a stick inside a swill bucket.
--- MultiMail/Win32 v0.49
* Synchronet * Freeway BBS in Bendigo, Australia.
* Origin: Vertrauen - [vert/cvs/bbs].synchro.net (1:103/705)

On 2017 Oct 27 06:56:00, you wrote to Digital Man:

Sounds like some really neat features in the pipeline. I'd like to be

able

to setup rules so echoes from a particular zone:net or a particular uplink went to a specific group, for example, new echoes from my feed in zone 21 would be created in my FSXNET group, and obviously those from my Fido uplink would go into the Fidonet group, etc.

hunh? you have only one fidonet group? we have at least five or six over here... something like this (from memory)...

FIBB - fidonet backbone
FIZ1 - fidonet zone 1 only
FISO - fidonet sysops only
FIZ1SO - fidonet zone 1 sysops only
FIR18 - fidonet R18 echos
FIR18SO - fidonet R18 sysops only
FIN3634 - fidonet N3634

lumping everything into one group is how leaks happen and trouble gets started when someone finds they have access to an area they should not have access to and they read something they should not have read... there are a lot more groups on the main system which is a backbone star and carries mail for other regions and nets... keeping areas' access properly deliniated is a major thing...

i agree that it would be nice to specify which group(s) new areas are to be added when they are sent automatically by a feed... we've been doing the autoadd of areas with fastecho since mud was invented... it is a good thing to see it finally being added to sbbsecho :)

)\/(ark

Always Mount a Scratch Monkey
Do you manage your own servers? If you are not running an IDS/IPS yer doin' it wrong...
... British Beef - The Recipe For hehheheheheh GAGAGAG WOOOOOOO!!!
---
* Origin: (1:3634/12.73)