Steam Chat room Chat logger

[ryodoan]

Chat logs found here: http://caffeinesystems.com/TF2Mappers

Well, to make this easy on everyone, I think from now on I will put the latest info up here in the top post.


My original hope when starting this project was that Valve would add Text logging onto Steam community rather rapidly, and therefore allow me to concentrate more on the Text->HTML aspect. As time has gone on and more problems arose with my Steam Chat Room->Text part of the equation the focus of the program shifted.

So, at the moment this is where the project stands. I would say I am on version 3 of the program, working on version 3.1.

Version 1:

• Was a simple, balanced match between AutoIt3 and VB.net coding.
• Very little logic, simple ran the AutoIt3 script every 30 minutes, then converted the text to HTML.
• Problems: Due to the lack of logic, it was high maintenance and not that stable. If anything did not happen as expected, it would get stuck.

Version 2:

• Was an attempt at creating a Database driven version of the program
• Used the same logging script as Version 1, but instead of rendering into HTML it inserted the data into a database.
• This database was then read by an ASP.net 2.0 program which could dynamically display the chat logs.
• Problems:
  • Just converting the entire first week of logging created over 28,000 rows in the messages table.
  • Its been 3+ years since I played with ASP.net and database driven websites, and I could not get a reliable website running
  • Did not have the time or motivation to figure out a user search for the program.
  • Hosting off my home computer + database data being sent over the internet + dynamically created pages = long loading times that where really, really annoying.
• Overall the final verdict was that I just did not have the time or motivation at this moment to do the job properly, so it has been put off.

Version 3

• Has been a near complete re-write of the code now that I have a better understanding of how everything is working.
• I have been able to trim down the AutoIt3 script to just the basics, this makes it faster and more reliable.
• The VB.net program handles creating new HTML pages for each day and appending HTML onto the existing HTML page for the current day.
• Extensive user options added making configuration MUCH easier where I had been using hard coded values that required re-compiling the program when something needed to be changed.

Version 3.1 (in the works)

• This is more of a continuation of 3 than any major changes.
• I am looking into adding FTP support to the program so I can move the HTML files automatically onto a hosted website, rather than running stuff off my personal web / file server.
• I noticed that in IE7 (have not checked 6) and when you have a lower resolution the navigation bar on the current log is not wide enough and ends up with a horizontal scroll bar, that is in the pipeline to be fixed.
• Check to see if steam Community is disconnected ("Your connection blah blah...") and if I am disconnected, kill steam + restart it.

So here is a detailed analysis of how exactly the program runs as of v3.

1. You start SteamLogger.exe
2. Timer starts counting down from 30 minutes (this is the default and can be changed in the settings)
3. Timer reaches 0 seconds
4. Program checks to see if there is a process named Steam.exe started
5. If the process is not found
   1. it runs an AutoIt3 script that launches steam (steam://run)
   2. then the script waits until steam has finished launching before exiting.
6. If the steam Launcher script has run for 60+ seconds, kill the steam launcher script, log an error, then go back to #2.
7. Run the steam Logger AutoIt3 script.
   1. Try to find the Steam Group Chat window (based off the set group name)
   2. If the Chat window is not open, open it
      1. This uses a handy dandy (steam://friends/joinchat/#######) command
   3. find the x, y for the chat window (top left corner)
   4. rt-click at x+15, y+115, aka just inside the top-left corner of the chat window area.
   5. click select all
   6. click copy
   7. close the chat window / open chat window (with the same handy steam://friends/joinchat/##### command)
      1. This is done to clear the chat window of any text.
   8. Exit the script
8. If the script has been running for 60+ seconds, kill the script, log an error, go back to step #2
9. Now, grab the text from the clip-board and store it into a string.
10. Check to see when the last log time was, if the new log time is on a different day than the last log time:
    1. Create a new HTML file based off the saved text.
    2. Update the navigation bar
    3. Save the new HTML files into the set directory
    4. Create / save a new text backup file in the set directory.
11. If the new log time is on the same day as the last log
    1. Read in the old HTML file, and trim off the ending tags.
    2. Render the saved chat log into HTML and append it to the end of the old HTML (along with the ending tags)
    3. Save the new HTML text into the old HTML file.
    4. Append the previous text backup with the new chat log text.
12. Set previousLogTime = current Log time
13. Return to step 2

So if we look at my original downfalls for the program:

1. The heart of the program is a script so it hijacks control of my server computer for intervals of about 4-5 seconds
   1. Yeah... unfortunately this is still valid and will remain valid until one of two things happen: 1. Steam implements a logging feature 2. Another logging method is found by me.
2. It requires steam to function perfectly for long periods of time
   1. Not so true anymore, I added some logic in to restart steam if it dies, and some other logic has gone into the AutoIt3 script to keep steam problems to a minimum. As I find a problem I write some code to handle it.
3. Its pretty specific to my computer setup.
   1. Not so true anymore either, I have added an extensive options panel to handle other computers with relative ease.
4. Logs are currently saved to my personal server therefore upload speeds will be slow and no guarantees are made to the servers uptimes (Unless I am not home and something goes wrong it should be up)
   1. Working on this one also with the FTP upload to my hosted web server.

If you have managed to read through all this, congratulations. I would upload a link to SteamLogger v3, except for a couple things.

1. It still needs to have a ReadMe written for it, believe me, you will need it. I have made the program MORE user friendly, but it still requires some setup (mostly how to set the settings).
2. I aint at home, and its not actually uploaded anywhere.

As always, if you are still interested despite the loggers posted flaws, you can contact me via PM on the forum or over Steam Chat. PM via the forums is probably the most reliable since I can check it from anywhere.


----- Old post below ------------
Well, I may not be a mapper, but instead of mapping I do other stuff. Like when someone suggests that it would be a good thing for us to be able to somehow reference old chats I got to thinking.

How hard would it be to write a chat logger?

So with trusty AutoIt3 at my side I wrote a simple script that goes to the chat window, copies all the text out of it, and then saves it to a text file.

Part B of this plan involves a companion program written in VB.net. The companion program runs the script every 30 minutes to grab the text and posts an alert in the chat every 60 minutes that the chat is being logged and where to find the chat log.

Features of the program include:

1. Saves all contents of the chat window every 30 minutes
2. Posts a link to the logs every 60 minutes along with an alert that the chats are being logged
3. Logs are re-written by the companion program into HTML with formatting.
4. Amazing page layout courtesy of Swift

There are a couple downfalls to the program:

1. The heart of the program is a script so it hijacks control of my server computer for intervals of about 4-5 seconds
2. It requires steam to function perfectly for long periods of time
3. Its pretty specific to my computer setup.
4. Logs are currently saved to my personal server therefore upload speeds will be slow and no guarantees are made to the servers uptimes (Unless I am not home and something goes wrong it should be up)

A copy of the chat logs can be found at: http://caffeinesystems.com/TF2Mappers

If anyone really wants a copy of the program / AutoIt script I dont have any problems sharing it but you will probably have to do some tweaking to get it to run properly on your own computer and unless you have a secondary computer that nobody is physically using I dont recommend using it because the program essentially will take over the user interface every 30 minutes.

p.s. Seriously thanks to Swift for help getting the CSS up and running.

 

[Alexforcefive]

Oh man, the log looks really nice now. Well done paul and swift!

 

[ryodoan]

I am still working on tweaking things on my end to make it so I can see more of what is happening and to make stuff a little more efficient, so the logs are being done more often than 30 minutes.

Errors fixed in no particular order:
- HTML tags typed in the chat window are now converted to web safe text
- Changed the font back to Arial
- The output.txt file is now saved as UTF-8 instead of ASCII so that the Save File part of the script does not get stuck again when there is a non-ascii character

 

[ryodoan]

An update.

TF2channel bot has a steam community page: https://steamcommunity.com/id/TF2mappers

I have started working on the logger V2.0 well really Log Web Interface 2.0. At the moment I use a program written in Visual Basic .net 2005 to generate a static HTML page from the saved text chat logs. This has several drawbacks:

• Page sizes get huge fast making for long download times.
• Searching is a pain and limited to Ctrl-F in browser
• Current methods for reducing page size (making a new webpage every day) are not very standard and not automated at all.

If you want to see the page develop as I work on it you can see my progress at: http://caffeinesystems.ath.cx/SampleASP/Default.aspx

Many changes are going to be on the backend at first and since I am trying to keep it as similar in appearance to the current log as possible you should not see a vast change in appearance.

I am about to fall asleep here typing so I am going to head to bed. Sorry if this post is at all incoherent but I am to tired to re-read it.

 

[Snipergen]

i read everything twice but i still dont get what your program does.

Does it save the text chat while playing? looks great though, i learn c++ my 2nd year at school but i suck in it, so , respect!

 

[ryodoan]

its fairly basic, here is a breakdown of what is going on.

The logger consists of two parts.
1. An AutoIt3 script that copies the chat room text into a text file.
2. A Visual Basic.net program that runs the script every 30 minutes and then converts the text log file into HTML.

Extreme breakdown:

• Manually open steam, my friends list, and the TF2mappers chat room.
• I start Chatter.exe my Visual Basic.net 2005 program.
• A timer starts counting down from 30 minutes
• When the timer hits 0 it launches an AutoIt3 script.
• AutoIt3 script activates the TF2mappers chat room, highlights all text in the window, copies it, , closes the Group chat window, opens the group chat window (this is done to clear all text from the window), starts notepad, opens my output.txt file, pastes new text into text file, closes and saves the output.txt file.
• Chatter.exe then reads in the old index.html file, chops off the ending tags (</div></body></html>), reads in output.txt and converts its contents to HTML, adds it onto the end of index.html, re-adds the closing tags and exports the new index.html file.
• Restarts the timer at 30 minutes.

My current project is to add a few steps in there that will take the output.txt file and save its contents into a database which will then be used by a dynamic website that reads the database when someone accesses the page. This makes it so I am no longer writing index.html. I would explain more but I have to get some work done...

At the moment this all runs on my server computer, but I have two more old machines sitting in my room doing nothing (a 800 mhz Celeron and a 400 mhz Pentium something-or-other), If I get the database stuff set up correctly this program will no longer need to run on my server and will be able to run it all from one of those computers.

 

[Saikan]

Damn this is good stuff man.

Paul deserves big props on this, he's spent more time getting this thing to work than most people do building nice looking TF2 maps.

 

[ryodoan]

Well since people are still reading this thread, I might as well keep it up to date with new happenings.

I have the database working fairly well right now, there are a couple small problems but nothing that can not be ironed out. I also have added onto my logging program so that it can now read the log output files and insert the messages into the database.

From running this method I now have over 27,000 rows in my messages table. Its a little higher than actual messages because it reads every new line in the chat log as a new message, so when someone sends a multi-line message it reads it as multiple messages.
ex:
"Scoobings: Raindrops keep falling on my head,
but that doesnt mean my eyes will soon be turning red.
Cryings not for me..."

Will be read as 3 messages, the bottom two as emotes by someone called "but" and "cryings" I think this is a fault of how I parse the text file, I need to see if the line breaks are at all different.

In any case, at the moment I am working on trying to figure out the best way to set up the entry page and how to best allow searching and displaying of the messages.

At the moment I show two calenders, you pick a start date in the first calender, and a stop date in the 2nd calender. Hit a button and it shows all messages between the two dates. The problem is that after every click the page refreshes causing a massively long reload time.

Here is a link to my current test page: http://caffeinesystems.ath.cx/SampleASP/demo.aspx WARNING THIS PAGE LOADS REALLY, REALLY SLOW and you need to wait for the entire page to load before clicking on ANYTHING ELSE or the page will crash.

Just to put it in perspective it took 12 seconds for the page to load at my work which has a really fast internet connection (my upload rate from home is only 80 KB/s)

 

[ryodoan]

*update 11/27/2007*
Stuff that is in the works now:

• http://caffeinesystems.ath.cx/SampleASP/demo.aspx now only has 1 calender on it. Click on a date, then click "Get messages" button. The resulting page may take a while to load and I would suggest not clicking on anything in the window again until the page finishes loading.
  - Ok, there was an error in my date creation, I fixed it but it may take some time to propagate to my server
• I have started working on a keyword search but am realizing the problems with writing my own. My plan is that by the end of the week there will be a simple keyword search where you type in some text and the program tries to find that EXACT text somewhere in the chats. So if you type in "con" It will find all messages with the words, "con, rcon, confession, and concord" if you type in "con cake" it will only return messages with "con cake in it" so "rcon cake is delicious" but not, "rcon is a strange way to mention cake". I am looking into a smart multi-keyword search but that requires more in depth SQL knowledge than I have right now.
• more search details: Basically you click search, it returns a list of messages that contain the word, you click on a message and it will display the previous 50 messages and the next 50 messages. (at least that is the plan at the moment and it is subject to change.

 

[TheBladeRoden]

I think I broke your calendar

Server Error in '/SampleASP' Application.
Invalid postback or callback argument. Event validation is enabled using <pages enableEventValidation="true"/> in configuration or <%@ Page EnableEventValidation="true" %> in a page. For security purposes, this feature verifies that arguments to postback or callback events originate from the server control that originally rendered them. If the data is valid and expected, use the ClientScriptManager.RegisterForEventValidation method in order to register the postback or callback data for validation.
Description: An unhandled exception occurred during the execution of the current web request. Please review the stack trace for more information about the error and where it originated in the code.

Exception Details: System.ArgumentException: Invalid postback or callback argument. Event validation is enabled using <pages enableEventValidation="true"/> in configuration or <%@ Page EnableEventValidation="true" %> in a page. For security purposes, this feature verifies that arguments to postback or callback events originate from the server control that originally rendered them. If the data is valid and expected, use the ClientScriptManager.RegisterForEventValidation method in order to register the postback or callback data for validation.

Source Error:

An unhandled exception was generated during the execution of the current web request. Information regarding the origin and location of the exception can be identified using the exception stack trace below.

Stack Trace:

[ArgumentException: Invalid postback or callback argument. Event validation is enabled using <pages enableEventValidation="true"/> in configuration or <%@ Page EnableEventValidation="true" %> in a page. For security purposes, this feature verifies that arguments to postback or callback events originate from the server control that originally rendered them. If the data is valid and expected, use the ClientScriptManager.RegisterForEventValidation method in order to register the postback or callback data for validation.]
System.Web.UI.ClientScriptManager.ValidateEvent(String uniqueId, String argument) +358
System.Web.UI.Control.ValidateEvent(String uniqueID, String eventArgument) +108
System.Web.UI.WebControls.Calendar.RaisePostBackEvent(String eventArgument) +49
System.Web.UI.WebControls.Calendar.System.Web.UI.IPostBackEventHandler.RaisePostBackEvent(String eventArgument) +7
System.Web.UI.Page.RaisePostBackEvent(IPostBackEventHandler sourceControl, String eventArgument) +11
System.Web.UI.Page.RaisePostBackEvent(NameValueCollection postData) +172
System.Web.UI.Page.ProcessRequestMain(Boolean includeStagesBeforeAsyncPoint, Boolean includeStagesAfterAsyncPoint) +4921


Version Information: Microsoft .NET Framework Version:2.0.50727.832; ASP.NET Version:2.0.50727.832

 

[ryodoan]

Does it do that every time you try to use the calender? I thought I had fixed that...

 

[Armadillo of Doom]

Excellent work my friend. I've been meaning to include this in my annoucements, but news has been kinda slow lately Still, I think this warrants it's own annoucement. Not many peeps visit the chat, so I don't think many know about the logger.

 

[Big Nick Digger]

any plans to release something like this for others?

 

[ryodoan]

any plans to release something like this for others?

I have a couple concerns with releasing it to other people:

Its a pretty primitive logger as in it will physically take over the computer when it logs, if you want to run it you have to have a machine that nobody is going to be wanting to use while you are logging.

There are 2 (3 with the latest version I am working on) very different parts to the logger.
1. The AutoIt3 script that copies text from the group chat window into a text file.
2. The VB.net Program that runs the AutoIt3 script and creates the HTML document (or in the new version inserts the messages into a database)
3. The ASP.net 2.0 website that reads the database and reacts to user input.

I would not really have a problem giving out the AutoIt3 script and the VB.net program that creates HTML, or the source code for either of those two parts, but I do not know anything about creating portable databases and all newer versions of my program are going to heavily rely on the SQL database on my server.

If you know much about coding I can give you the AutoIt3 script which creates the Text Log and then you could probably figure out what you want to do form there.

Send me a PM over the forum if you are interested and we can probably work something out.

P.S I noticed on http://caffeinesystems.ath.cx/TF2Mappers that I linked to index10.html twice, the 2nd link should have gone to http://caffeinesystems.ath.cx/TF2Mappers/index11.html . I will probably fix it this evening.

 

[ryodoan]

I must confess I have not done much work on the logger other than to fix a small bug that was causing it to die after receiving an invite. I have been working on coding at real work and when I get home I have just been sluming around to tell the truth.

In any case, this post is to let you guys see what the javascript on the page has been doing.

Its a map of all yall who have visited the logger site, and one of the inspirations I have to get this thing actually working. No need to be paranoid, this map is generated by your IP address which can actually be mapped out to your ISP.

Here is a really page that you can see some sort of an explanation, if its not an explanation, then forgive me because I am really tired and this was my first thought:

http://xkcd.com/195/

Now... time for bed

 

[ryodoan]

Ok, a long, long delayed reply and update to this thread.

To start off with I have pretty much decided that a database driven version is not the best idea for a few reasons:

1. Huge amount of data + poor search algorithm = crap
2. If I leave this as static HTML google can index it for me
3. Huge amount of data + crappy computer + bad internet upload speeds = my computer crapping its pants (the first week of data was over 27,000 entries into the messages table)
4. I dont know php + mysql, and while I have a real website host, they dont support asp.net + SQL server.
5. Was more work than I bargained for and I dont have the time to properly finish it. Oh, and other people were getting strange errors...

That is just a few of the issues I was having with the database backend. So my new plan that I have started to implement is this:

1. Re-write the logging program to be more reliable and autonomous.
   • I have already started this. I have already made it so that the 3 biggest problems should be fixed.
     • AutoIt3 freaking out when opening notepad, launching the logger into a cycle of death getting stuck launching notepad. I didnt catch it for 24 hours one day and had 48 copies of notepad opened on my computer and 48 copies of the autoit3 script running stuck in the same place.
       • I dont use notepad anymore, instead grab the text straight from the clipboard after it has been copied.
     • If Steam died somehow, the logger would obviously stop working.
       • Fixed by creating a new script dedicated to launching steam.
     • This was a problem that cropped up from problems 1 and 2, basically if something unexpected happens during the logging process, the script gets stuck, and every log interval a new script is launched, and gets stuck.
       • Added logic to kill scripts after they have been running for 60 seconds. (The actual logging should take 10 seconds max, so if its longer than that there is a problem)
   • More autonomous in that I want it to be able to create new daily HTML pages and update a central page with links to the daily logs. At the moment it just keeps building an ever bigger html page until I think its to big and swap to a new one.
2. By using static HTML pages I can probably add a google search bar to the website and let it index the logs. This will give us a much more powerful search that will help you guys find stuff easier.

here is an example picture I whipped up of what I am thinking the new homepage will look like:

Is this a good idea? or would you prefer that the most recent logs stay on the front page? AHA, just had a thought, I can just embed the logs below in a separate frame, yay! that makes life easy for me, hehe.

Oh, and this thread was first made before the TF2mappers group had its own forum, so would a moderator / admin please move it there?

 

[ryodoan]

just to keep a running commentary going here.

First, I noticed a lot of new traffic coming from this thread and I just wanted to make sure it was clear that the new logger is not finished yet and therefore you are not going to see any changes in the chat logger for a while yet. I would say it is about at 50% finished. I have finished the basis for the options menu (stuff keeps getting added to it as I go), it can run the scripts, check for their completion, pause the program while it waits for them, and correct them as needed, and then read in the info from the clipboard, then convert that read in text into HTML, but I have not finalized how it writes those HTML files yet. I am working on hammering out how exactly I want the HTML index formatted (important because I also want this page to be generated / updated from code).

So, my current part of the project is two fold. 1. Figure out how I want the index to look, 2. Figure out how to generate the required HTML files.

for part 1, I am thinking I am going to come forward from 1990 to 1994 and implement a frame based navigation bar. So that no matter which log you are currently viewing, you will have a sidebar with links to all the other logs, the question I am left with is how to format that side bar.

Part 2 requires Part 1 to be finished, and I have never claimed to be any sort of a good web designer, on the contrary, I know I suck at it. Swift helped me out with the current design but it appears he has moved onto bigger, better things. (His website is down and he has not been in the chat since 12/18/07 (yes I used the logs to find that ))

 

[Squirrelly]

The logger is watching your every move and every word, be aware!

very cool feature & good work!

 

[ryodoan]

Another update here:

So at the moment this is what the design looks like. This is about 75% output from the program, then I did some touch up work because the output was as bit screwy.

Some things to keep in mind are that the design is not final yet, this is mostly a mock-up where I opened it up in notepad and typed in some extra stuff to tweak it around. I am still thinking about different ways to format the links, maybe have a steam-esque type image behind them and different font / formatting for them, and look into making the frame less noticeable.

I have said it before and I will say it now, I am pretty miserable at making this look really good, but I want it to at least look tolerable

Oh, also, all times for the new log will be based off GMT, I figure since we are a world wide group its kinda unfair for me to have all the logs dated from EST only. So in future logs all times / dates will be based off GMT and you can figure out your local time from that.

 

[ryodoan]

HUGE NEW DEVELOPMENT OF CRAZY AWESOMENESS!!!

Lets start with some history (of the chat logger). If you read all my posts here you know that the Logger has 2 parts. Part 1 is a VB.net 2005 program, Part 2 is an AutoIt3 script. The script is the biggest weakness in this program because for it to work, stuff has to be in the place it remembers it being. I try to compensate for that by making it use Window placement depended coordinates (rough example: FindWindow("windowName").Xpos + 15 ) But, if the control I want to click at Xpos + 15 for some reason is now at Xpos + 30, I am SOL and the script locks up as it waits for some action to happen, that doesnt happen. Because of this problem, I have been trying to work as far away from AutoIt3 as possible, and put more dependence on the VB.net program.

So here is how the AutoIt3 logger worked when I first started:

1. Check to make sure steam is open
   1. If not, exit script
2. Check to see if Steam Friends is open
   1. If not, open it.
3. Check to see if Steam Group Chat is open
   1. If not, run the openChat() function (described later)
4. Activate the Steam Group Chat window
5. Click+hold down button of mouse in lower right corner
6. Send Home key (to highlight all text)
7. Release mouse button
8. send Ctrl-C to window
9. Close Group chat window
10. Open Group chat window with openChat() function
    1. OpenChat() function
       1. Find X,Y of friends window
       2. Click at x+100, y+100 to get the group menu
       3. Click at X+105, y+115 to join group chat
       4. Note, this requires the user to have at least 1 friend, and the friends "+" list to be closed.
11. Open Notepad
12. Activate Notepad Window
13. Send ctrl-o to get the "Open File Dialog"
14. Wait for Open File DIalog
15. send "output.txt" to Open File Dialog
16. Send "{enter}" to Open File Dialog
17. Wait for "Output.txt - Notepad" window to be active
18. Send Ctrl-V
19. Send Ctrl-S
20. Close Notepad.

As you can see, its a long process with a TON of places it can go wrong. So, with my current version of the script that I am working on, I managed to cut steps 11-20 out of the loop, and I made steps 5-9 much simpler. But the next biggest, if not BIGGEST problem in the entire loop, is the "openChat()" It depends the most on relative mouse clicks and your friends list being just exactly right. If any one thing is different, the opening of the group chat window will fail, and the logger will be stuck.

One of the checks that I just put into the VB.net part of the program, is it checks to see if steam is running, then starts it. To do this, I used the "steam://run" function. I thought there must be a way to launch the group chat from this, but I couldnt find documentation on it online.

Then, while I was sitting here at work, waiting for a program to finish running, I thought, "WAIT!!! you can open the steam group chat FROM THE STEAM COMMUNITY PAGE." I jumped over to the Steam Community page for our group, and found this gem:

steam://friends/joinchat/103582791429594873

Now, what would make this all deluxe, is if I can look up how to get this: "103582791429594873" programmatically. Some other people online have figured it out, and written a PHP API for grabbing that kinda info.

Here is a demo of that API: http://seirmubsa.com/?g=steamsig

*edit* well, I flipped through their PHP code and it seems that they are just going to the steam community website, and parsing the Steam Community ID # out of the HTML. Since I only want the group chat it should be a lot easier for me.