Hello Digital,

Okay, so apparantly when CRC checking is disabled, MSGID checking is done. But when CRC checking is enabled, MSGID checking is disabled? I would assume that CRC checking would be an additional feature to the original dupe checking. But this doesn't seem to be the case.

Regards,
Nick

... "-Y-| -+-+-#-A. -> -+-|-|-U-i -e-+-+-i-|-+ -C-#-#-+-e-#-A."
--- GoldED+/LNX 1.1.5-b20161221
# Origin: thePharcyde_ distribution system (Wisconsin) (723:1/1)
* Synchronet * thePharcyde_ telnet://bbs.pharcyde.org (Wisconsin)
* Origin: Vertrauen - vert.synchro.net (1:103/705)

Re: Not sure what's going on..
By: Accession to Digital Man on Fri Feb 24 2017 10:44 pm

Hello Digital,

Okay, so apparantly when CRC checking is disabled, MSGID checking is done. But when CRC checking is enabled, MSGID checking is disabled?

No, duplicate message-ID checking is *always* performed on imported messages (but not for pass-through areas).

I would
assume that CRC checking would be an additional feature to the original dupe checking. But this doesn't seem to be the case.

It is.

digital man

Synchronet/BBS Terminology Definition #16:
DOVE = Domain/Vertrauen
Norco, CA WX: 61.0oF, 42.0% humidity, 0 mph ENE wind, 0.00 inches rain/24hrs --- SBBSecho 3.00-Win32
* Origin: Vertrauen - vert.synchro.net (1:103/705)

Hello Digital,

On Sat Feb 25 2017 12:30:56, Digital Man wrote to Accession:

Okay, so apparantly when CRC checking is disabled, MSGID checking
is done. But when CRC checking is enabled, MSGID checking is
disabled?

No, duplicate message-ID checking is *always* performed on imported messages (but not for pass-through areas).

By "pass-through" areas, do you simply mean areas you get from a feed, and feed

to someone else? If so, why would you want to disable MSGID checking in that case?

I would
assume that CRC checking would be an additional feature to the
original dupe checking. But this doesn't seem to be the case.

It is.

Something doesn't seem to jive with this information then. When using CRC checking Fidonet monthly robot postings (which are the same, but with a different MSGID each time) seem to be caught as dupes when they're not. Disabling CRC checking seems to fix this issue, so it would seem also that either CRC checking is overriding the MSGID check and catching the message as a

dupe before the MSGID is checked, or is disabling the MSGID check altogther. Either way, it's not being noticed that it is a new message since the MSGID is different, and the date/timestamp is different.

Regards,
Nick

... "-Y-| -+-+-#-A. -> -+-|-|-U-i -e-+-+-i-|-+ -C-#-#-+-e-#-A."
--- GoldED+/LNX 1.1.5-b20161221
# Origin: thePharcyde_ distribution system (Wisconsin) (723:1/1)
* Synchronet * thePharcyde_ telnet://bbs.pharcyde.org (Wisconsin)
* Origin: Vertrauen - vert.synchro.net (1:103/705)

Re: Not sure what's going on..
By: Accession to Digital Man on Sat Feb 25 2017 06:20 pm

Hello Digital,

On Sat Feb 25 2017 12:30:56, Digital Man wrote to Accession:

Okay, so apparantly when CRC checking is disabled, MSGID checking
is done. But when CRC checking is enabled, MSGID checking is
disabled?

No, duplicate message-ID checking is *always* performed on imported messages (but not for pass-through areas).

By "pass-through" areas, do you simply mean areas you get from a feed, and feed
to someone else?

No, pass-through areas are areas in your area file (areas.bbs) which do not have a local persistent-storage message base (a 'P' is specified for the msg base internal code).

If so, why would you want to disable MSGID checking in that case?

Pass-through areas don't have anywhere to store messsage-IDs (since there is no local msgbase).

I would
assume that CRC checking would be an additional feature to the
original dupe checking. But this doesn't seem to be the case.

It is.

Something doesn't seem to jive with this information then. When using CRC checking Fidonet monthly robot postings (which are the same, but with a different MSGID each time) seem to be caught as dupes when they're not. Disabling CRC checking seems to fix this issue, so it would seem also that either CRC checking is overriding the MSGID check and catching the message as a
dupe before the MSGID is checked, or is disabling the MSGID check altogther. Either way, it's not being noticed that it is a new message since the MSGID is different, and the date/timestamp is different.

The duplicate message body text checking is in *addition* to MSG-ID checking. If you have duplicate message body checking enabled (which uses multiple methods, not just CRC) and a duplicate message is dedicted using that method, it doesn't matter that the MSG-IDs might be different, it's still considered a dupe.

digital man

Synchronet "Real Fact" #38:
Synchronet first supported Windows NT-based operating systems w/v3.00b (2000). Norco, CA WX: 48.7oF, 84.0% humidity, 4 mph SE wind, 0.00 inches rain/24hrs
--- SBBSecho 3.00-Win32
* Origin: Vertrauen - vert.synchro.net (1:103/705)

Hello Digital,

On Sat Feb 25 2017 22:00:22, Digital Man wrote to Accession:

No, pass-through areas are areas in your area file (areas.bbs) which
do not have a local persistent-storage message base (a 'P' is
specified for the msg base internal code).

Okay. Since that's news to me I have never done it this way. There has always been a persistant storage in a message base.

Something doesn't seem to jive with this information then. When
using CRC checking Fidonet monthly robot postings (which are the
same, but with a different MSGID each time) seem to be caught as
dupes when they're not. Disabling CRC checking seems to fix this
issue, so it would seem also that either CRC checking is overriding
the MSGID check and catching the message as a dupe before the MSGID
is checked, or is disabling the MSGID check altogther. Either way,
it's not being noticed that it is a new message since the MSGID is
different, and the date/timestamp is different.

The duplicate message body text checking is in *addition* to MSG-ID checking. If you have duplicate message body checking enabled (which
uses multiple methods, not just CRC) and a duplicate message is
dedicted using that method, it doesn't matter that the MSG-IDs might
be different, it's still considered a dupe.

Then this seems to be what's happening, but also seems to be causing false positives. Monthly robot postings, and for example the monthly posting of the Fidonews or Fidogazette publications are posted monthly with new MSGIDs and new

date/timestamps. However, these robot postings and some of the final pages of both publications are caught as dupes because the message body text does not change (even though date/time and MSGID does).

As I mentioned prior, disabling any CRC checking seems to change the outcome of

this, but if someone wants to strengthen their dupe checking they shouldn't be worried about false positives being caught and removed. I would think the the MSGID *and* the duplicate message body text should work together, rather than overriding the fact that the MSGID is different.

Regards,
Nick

... "-Y-| -+-+-#-A. -> -+-|-|-U-i -e-+-+-i-|-+ -C-#-#-+-e-#-A."
--- GoldED+/LNX 1.1.5-b20161221
# Origin: thePharcyde_ distribution system (Wisconsin) (723:1/1)
* Synchronet * thePharcyde_ telnet://bbs.pharcyde.org (Wisconsin)
* Origin: Vertrauen - vert.synchro.net (1:103/705)

Re: Not sure what's going on..
By: Accession to Digital Man on Sun Feb 26 2017 07:30 am

Hello Digital,

On Sat Feb 25 2017 22:00:22, Digital Man wrote to Accession:

No, pass-through areas are areas in your area file (areas.bbs) which
do not have a local persistent-storage message base (a 'P' is
specified for the msg base internal code).

Okay. Since that's news to me I have never done it this way. There has always been a persistant storage in a message base.

Something doesn't seem to jive with this information then. When
using CRC checking Fidonet monthly robot postings (which are the
same, but with a different MSGID each time) seem to be caught as
dupes when they're not. Disabling CRC checking seems to fix this
issue, so it would seem also that either CRC checking is overriding
the MSGID check and catching the message as a dupe before the MSGID
is checked, or is disabling the MSGID check altogther. Either way,
it's not being noticed that it is a new message since the MSGID is
different, and the date/timestamp is different.

The duplicate message body text checking is in *addition* to MSG-ID checking. If you have duplicate message body checking enabled (which uses multiple methods, not just CRC) and a duplicate message is dedicted using that method, it doesn't matter that the MSG-IDs might
be different, it's still considered a dupe.

Then this seems to be what's happening, but also seems to be causing false positives. Monthly robot postings, and for example the monthly posting of the Fidonews or Fidogazette publications are posted monthly with new MSGIDs and new
date/timestamps. However, these robot postings and some of the final pages of both publications are caught as dupes because the message body text does not change (even though date/time and MSGID does).

That's how it is intended to work. If you want to have the same message body repeated in your message base(s), then disable duplicate message checking in them.

As I mentioned prior, disabling any CRC checking seems to change the outcome of
this, but if someone wants to strengthen their dupe checking they shouldn't be worried about false positives being caught and removed.

It's not a "false positive". If the message has the exact same message text, then it's a duplicate.

I would think
the the MSGID *and* the duplicate message body text should work together, rather than overriding the fact that the MSGID is different.

I disagree. It's completey feasible that old messages could accidentally or maliciously enter a network with newly-generated or modified message-IDs.

digital man

Synchronet/BBS Terminology Definition #50:
Sysop = System Operator
Norco, CA WX: 52.5oF, 83.0% humidity, 0 mph WNW wind, 0.11 inches rain/24hrs --- SBBSecho 3.00-Win32
* Origin: Vertrauen - vert.synchro.net (1:103/705)

Hello Digital,

On Sun Feb 26 2017 12:16:18, Digital Man wrote to Accession:

That's how it is intended to work. If you want to have the same
message body repeated in your message base(s), then disable duplicate message checking in them.

Okay. I didn't understand it this way the first time you explained it to me. Now I do.

I disagree. It's completey feasible that old messages could
accidentally or maliciously enter a network with newly-generated or modified message-IDs.

Good point, and it does indeed happen (even recently).

Regards,
Nick

... "-Y-| -+-+-#-A. -> -+-|-|-U-i -e-+-+-i-|-+ -C-#-#-+-e-#-A."
--- GoldED+/LNX 1.1.5-b20161221
# Origin: thePharcyde_ distribution system (Wisconsin) (723:1/1)
* Synchronet * thePharcyde_ telnet://bbs.pharcyde.org (Wisconsin)
* Origin: Vertrauen - vert.synchro.net (1:103/705)

Re: Not sure what's going on..
By: Digital Man to Accession on Sun Feb 26 2017 12:16 pm

I disagree. It's completey feasible that old messages could accidentally or maliciously enter a network with newly-generated or modified message-IDs.

Rob,

What about the case where a moderator of an echo *MUST* post the rules of the echo in it. Many times
there are robo-posts that contain the exact same text but are posted.

In the case of the fidonews echo there are some pages that are the same like for example, page 6 which
lists the current version numbers and where to download BBS software. Sometimes there are several months
with no change, so the page is repeated. Those are not coming through.

Most softwares that we see use MSGID + MSGBODYCRC to determine a dupe. If the MSGID's match that does
not necessarily mean that the message areas are the same either. In fact there
has been cases where two
differnt systems generate the same MSGID.

Fred Riccio has a three phase dupeck program. It uses MSGID + MSGHEADERCRC + MSGBODYCRC to must match
before it calls it a dupe.

Quite frankly, I like having the robomessages negated, But it would be great to have a little closer
checking on the duplicate or at least have configurability on how the duplicate
message would be
determined.

Allen
--- SBBSecho 3.00-Win32
* Origin: LiveWire BBS - Synchronet - LiveWireBBS.com (1:2320/100)

On 2017 Feb 27 21:42:44, you wrote to Digital Man:

In fact there has been cases where two differnt systems generate the
same MSGID.

this cannot happen... the MSGID is composed of the address and the serial number,,, it takes both, together, to comprise a MSGID... serial numbers /can/ be duplicated from different addresses... it is the combination of address and serial number that cannot be duplicated within a three year period...

)\/(ark

Always Mount a Scratch Monkey
Do you manage your own servers? If you are not running an IDS/IPS yer doin' it wrong...
... There's no FAT in a 0 byte file.
---
* Origin: (1:3634/12.73)

Re: Not sure what's going on..
By: mark lewis to Allen Prunty on Mon Feb 27 2017 10:28 pm

this cannot happen... the MSGID is composed of the address and the serial number,,, it takes both, together, to comprise a MSGID... serial numbers /can/ be duplicated from different addresses... it is the combination of address and serial number that cannot be duplicated within a three year period...

I know of one software that's out there that generates MSGID's using sequential numbers.

There are very RARE instances where it is possible.

Allen
--- SBBSecho 3.00-Win32
* Origin: LiveWire BBS - Synchronet - LiveWireBBS.com (1:2320/100)

On 2017 Feb 27 23:09:18, you wrote to me:

this cannot happen... the MSGID is composed of the address and the
serial number,,, it takes both, together, to comprise a MSGID... serial
numbers /can/ be duplicated from different addresses... it is the
combination of address and serial number that cannot be duplicated
within a three year period...

I know of one software that's out there that generates MSGID's using sequential numbers.

that's perfectly fine as long as the counter doesn't reset... that was part of the problem with at least one sysop and his experiments and reinstallations...

There are very RARE instances where it is possible.

you've missed what i said, allen... reread your original statement...

In fact there has been cases where *two differnt systems* generate the same MSGID.

notice the hilite i placed... two different systems cannot generate the same MSGID because their addresses are different... if you had said "same MSGID serial number" i would fully have agreed but would still have pointed out that that is ok because of the different addresses...

)\/(ark

Always Mount a Scratch Monkey
Do you manage your own servers? If you are not running an IDS/IPS yer doin' it wrong...
... I'm not bossy. I just know what you should be doing.
---
* Origin: (1:3634/12.73)