Handle dash for xff, and region id starting the path #712
Conversation
| @@ -119,6 +120,16 @@ Resources: | |||
| }); | |||
| }; | |||
|
|
|||
| const findIp = (xff, ip) => { | |||
| if (xff === '-') { | |||
There was a problem hiding this comment.
This is the bug that made all the hashed ips the same - the xff is coming through most of the time as -, which is not blank, but also not an ip. This dash was being used as the ip instead of the client ip
There was a problem hiding this comment.
Oof - good catch. I think I had similar in the counts-lambda, but apparently forgot about it here.
| @@ -146,6 +157,12 @@ Resources: | |||
| // podcast id and episode guid (only works for dovetail3-cdn requests) | |||
| const datas = mappedRows.filter(data => { | |||
| const parts = data['cs-uri-stem'].split('/').filter(s => s); | |||
|
|
|||
| // if the path starts with a region like usw2, shift that off | |||
| if (parts[0] && parts[0].match(/^[a-z][a-z0-9\-]+$/)) { | |||
There was a problem hiding this comment.
This was the other bug, that we have requests with an aws region name prefix, like /usw2/
These were all getting filtered out
| @@ -163,8 +180,7 @@ Resources: | |||
| // calculate listener_ids | |||
| datas.forEach(data => { | |||
| // use leftmost XFF or IP | |||
| const xffParts = (data['x-forwarded-for'] || '').split(',').map(s => s.trim()).filter(s => s); | |||
| const leftMostIp = xffParts[0] || data['c-ip']; | |||
There was a problem hiding this comment.
this comparison was picking the dash, -, value of the xff over the client ip
| @@ -119,6 +120,16 @@ Resources: | |||
| }); | |||
| }; | |||
|
|
|||
| const findIp = (xff, ip) => { | |||
| if (xff === '-') { | |||
There was a problem hiding this comment.
Oof - good catch. I think I had similar in the counts-lambda, but apparently forgot about it here.
| @@ -146,6 +157,12 @@ Resources: | |||
| // podcast id and episode guid (only works for dovetail3-cdn requests) | |||
There was a problem hiding this comment.
Not related directly to this change but ...
For re-processing purposes, it may be useful to also have this lambda log what S3 input file it's processing, and how many rows it had. Just above this line somewhere:
console.info(`Read ${rows.length} rows from s3://${Bucket}/${Key}`);
fixes #714