Hi there! You are currently browsing as a guest. Why not create an account? Then you get less ads, can thank creators, post feedback, keep a list of your favourites, and more!
Not actually evil.
#101 Old 17th Jul 2009 at 8:13 PM
You can compress/decompress with SimPE or JFade's Compressorizer (or my stuff). Make two versions of a record, compressed and uncompressed, with SimPE or another program, then try the same thing with your code, and see where your version differs from SimPE's (or whatever's). Binary data is easier to compare if you use a hex editor.

Please spay or neuter your pets. --- Cat Music Video! --- my meshes
Advertisement
Warrior Gryphon
site owner
#102 Old 17th Jul 2009 at 8:17 PM
Cat: It'd probably be easier to do side by side comparisons of the actual algorithms though.

Story books are full of fairy tales, of Kings and Queens, and the bluest skies.
Mad Poster
Original Poster
#103 Old 17th Jul 2009 at 8:54 PM Last edited by GeneralOperationsDirector : 17th Jul 2009 at 9:36 PM.
Code:
  @staticmethod
  def decompress(ExpectedSize, Data):
    print "Attempting decompression."
    import array
    SourceData = array.array("B", Data)
    SourceLength = len(SourceData)
##    CompressionHeader = struct.unpack("<I")
    ReturnLength = struct.unpack("<I", SourceData[:4].tostring())[0]
    print ExpectedSize, ReturnLength
    ReturnData = array.array("B", (0,))*ExpectedSize
    SourceIndex, ReturnIndex = 9, 0
    try:
      while SourceIndex < SourceLength:
        # Opcode is the code for the decompression operation to perform.
        # Count1 is the number of plain-text characters to copy from the source stream to the destination stream.
        # Count2 is the number of plain-text characters to copy from the DESTINATION stream to the end of the destination stream.
        # Offset is the end-relative location of the start of the characters to copy from the DESTINATION to the end of the destination.
        Opcode = SourceData[SourceIndex]
        Count1 = Count2 = Offset = 0
        if 0x00 <= Opcode <= 0x7F:
          Count1 = (Opcode & 0x03)
          Count2 = ((Opcode & 0x1C) >> 2) + 3
          Offset = ((Opcode & 0x60) << 3) + SourceData[SourceIndex+1]
          print hex(Opcode), Count1, Count2, Offset
          SourceIndex += 2
        elif 0x80 <= Opcode <= 0xBF:
          Count1 = (SourceData[SourceIndex+1] & 0xC0) >> 6
          Count2 = (Opcode & 0x3F) + 4
          Offset = ((SourceData[SourceIndex+1] & 0x3F) << 8) + SourceData[SourceIndex+2]
          print hex(Opcode), Count1, Count2, Offset
          SourceIndex += 3
        elif 0xC0 <= Opcode <= 0xDF:
          Count1 = (Opcode & 0x03)
          Count2 = ((Opcode & 0x0C) << 6) + 5 + SourceData[SourceIndex+3]
          Offset = ((Opcode & 0x10) << 12) + (SourceData[SourceIndex+1] << 8) + SourceData[SourceIndex+2]
          SourceIndex += 3
        elif 0xE0 <= Opcode <= 0xFC:
          Count1 = (Opcode - 0xDF) << 2
          print hex(Opcode), Count1
          SourceIndex += 1
        else:
          print hex(Opcode)
          print repr(ReturnData[:ReturnIndex].tostring())
          assert False, "Unknown opcode"
        if Count1: # If we have any new plain-text characters this iteration, this is where we copy them. Note that this is a bulk-copy.
          print repr(SourceData[SourceIndex:SourceIndex+Count1].tostring())
          ReturnData[ReturnIndex:ReturnIndex+Count1] = SourceData[SourceIndex:SourceIndex+Count1]
          SourceIndex += Count1
          ReturnIndex += Count1
        if Count2: # If we have any OLD plain-text characters this iteration, this is where we copy them. Note that this is a character-by-character copy.
          for i in range(0, Count2):
            ReturnData[ReturnIndex + i] = ReturnData[ReturnIndex-Offset-1 + i]
          print repr(ReturnData[ReturnIndex-Offset-1:ReturnIndex-Offset-1+Count2].tostring())
          ReturnIndex += Count2
    finally: 
      # The try-finally statement is only for debugging purposes, so that the decompressed data can be recovered conviently.
      # The try statement and the entire finally clause will vanish in production, but the content of the try clause will remain,
      # as that is what is being debugged.
      print "An error has occured. Final content of decompression buffer:"
      print repr(ReturnData[:ReturnIndex].tostring())
    return ReturnData.tostring()

This Space Intentionally Left Blank
Not actually evil.
#104 Old 17th Jul 2009 at 8:57 PM
Quote: Originally posted by Delphy
Cat: It'd probably be easier to do side by side comparisons of the actual algorithms though.

I thought G. meant he had the algorithm figured out and was asking for help finding where his output starts being messed up, so he could hunt down the bug(s) in his source.

It would be useful to look at the algorithm, of course, so we know if given a certain input, what the correct output should be, and how that output is produced. But once we do that, we need to know that the implementation of the algorithm is correct, and that's what I thought he was asking about.

G, I assume you're using pydebug, IDLE, or some other python debugger / IDE with breakpoints and watchwindows? (and the aforementioned hex editor)

edit: Delphy, did you mean look at G's python code next to your PHP? I'm python illiterate, and usually refuse to look at anyone's code line-by-line, so I wouldn't be much help with that. I'm happy to discuss pseudocode and debugging.

Please spay or neuter your pets. --- Cat Music Video! --- my meshes
Mad Poster
Original Poster
#105 Old 17th Jul 2009 at 9:00 PM
I develop with IDLE, and I have SimPE. I tried to post the source of the relavent definition, but something is screwing with the indentation, and indentation is IMPORTANT for Python source. I`m still messing with that post.

Edit: The whitespace *is* preserved when I go back to edit the post; it just isn`t displayed!

Edit again: Bah! even the PRE tag doesn`t help! Gah!

Edit again: The CODE tag does it, but it DOUBLE SPACES the lines, which is harmless for Python, but makes it difficult to read. Going back to delete the periods at the beginnings of the lines.

Edit again: Well, I guess that`ll have to do. It seems to be the best I can do. If anyone wants to see the package I`m attempting to decompress, the NAME of the file is visible in the debugging output.

This Space Intentionally Left Blank
Not actually evil.
#106 Old 17th Jul 2009 at 9:11 PM
G, your python copy and paste has lost all of its formatting and indentation, try .... for spaces. fixed

I notice a complete lack of comments in that code. Nothing that says "read header", "this is the uncompressed size", "store a repeating string", "write the location of next string occurence" or anything like that. Make this easier on yourself and anyone willing to read other people's code, and please, please comment.

edit: actually, the double spacing makes it much easier for me to read, not that I will since I don't go over people's code line by line (could make exception if there were proper comments)

Please spay or neuter your pets. --- Cat Music Video! --- my meshes
Mad Poster
Original Poster
#107 Old 17th Jul 2009 at 9:26 PM
I *try* to choose meaningfull names, so that the lack of comments isn`t burdensome, but I can go back and add some to my post, if you like. I`m also expecting ExpectedSize and ReturnLength to be equal, but they aren`t. Obviously, I don`t cover the compleate range of opcodes, but it should correctly handle the range it does cover, and I added the opcodes as I encountered them during testing.

Oh, there is no "read header" because the raw data is read elsewhere and passed into the function as a parameter, and because my attempts to parse the compression header were less than successfull. I went ahead without parsing the header, though, because I found the information allegedly contained in that header to be available elsewhere: ExpectedLength from the compression index, SourceLength from the main index, and only one compression method to chose from. The other two suggested comments don`t seem to match anything the code does. The code is structurally identical to Delphy`s PHP code, as that was the easiest way to translate his code. Might that help any?

This Space Intentionally Left Blank
Not actually evil.
#108 Old 17th Jul 2009 at 9:35 PM
Meaningful variable names are great, but they don't make up for lack of comments. Look at that code a year from now and see if you think it's still immediately clear what it does.

If you don't parse the compression header, do you write out a correct compression header? Other programs use those headers. That info is elsewhere, yes, but one should check the header too, make sure you don't have a bad resource or corrupted file.

Please spay or neuter your pets. --- Cat Music Video! --- my meshes
Mad Poster
Original Poster
#109 Old 17th Jul 2009 at 9:40 PM
I don`t write out ANYthing, yet! I wouldn`t DARE write out what I cannot yet READ!

Not to worry, though: I will make a point of writing the decompression header correctly, when I`m ready to start trying to write things.

Actually, that will be easy: I simply will not compress ANYthing. No compression, no compression header, therefore no BAD compression headers. I have never indended to write compressed data at all, EXCEPT when copying compressed data from a source package to a target package, where I will simply trust that the existing header is correct. Will this do? If the resulting package NEEDS to be compressed, jfade`s Compressorizor will do nicely, nyet?

As for comments, I tend to add that kind of documentation later, after I get the code *working*, though sometimes I document things that are KNOWN to work poorly, to help me remember what the deficiency is when I come back later.

Gotta go! Late for work; see ya next week!

This Space Intentionally Left Blank
Not actually evil.
#110 Old 17th Jul 2009 at 9:50 PM
Quote: Originally posted by GeneralOperationsDirector
Actually, that will be easy: I simply will not compress ANYthing. ... If the resulting package NEEDS to be compressed, jfade`s Compressorizor will do nicely, nyet?

Da. Just let folks know the output is not compressed, and they'll know to compress it if they want to.

Please spay or neuter your pets. --- Cat Music Video! --- my meshes
Mad Poster
Original Poster
#111 Old 20th Jul 2009 at 7:21 PM
Hokay.

Meanwhile, thanks to your help last week, I was, over the weekend, able to find and FIX the error. I used SimPE to view the correctly-decoded data, and discovered that it didn`t match my data AT ALL. I then started adding additional debugging output, and discovered that the program was mysteriously opening a file that apparently had nothing to do with the file I was attempting to open. THAT struck me as "insane" behavior, but I eventually tracked it down to a typo that was causing the program to open "D:\\...." instead of "D:\...."; apparently the extra backslash was wreaking all kinds of oddness with the file-open code.

Once THAT was dealt with, I discovered that the GLOB in the file I *thought* I was processing was not compressed at all, so I deliberately sought out and processed the file that I had been accidentally processing earlier, because I still had an error in the decompression code.

Once THAT was delt with, I was finally able to compare correctly-processed data with incorrectly-processed data from the same file, and discovered that I was right about the error being one of the off-by-one variety, in this case in the last opcode processed, which was not being displayed in the debugging output because of bugs in the debugging code. Sheesh. The processing of the opcode was included in the output, but the code to display the opcode itself had been inadvertantly omitted.

The actual error in the processing that was causing the decompression code to throw index-out-of-range errors for bogus opcodes was adding one less than needed to account for the length of the opcode whose display [but not processing] was accidentally left out. ANY processing after encountering that opcode, including that for the opcode itself, would have been faulty, but the nature of the failure depended upon the specific data being misinterpreted.

In a nutshell, your suggestion last week did the trick. Thank you VERY much!

As for checking the compression header for corruptedness, I`d already thought of that, but was more concerned short-term in getting the exanguinating thing exanguinated working, si?

This Space Intentionally Left Blank
Not actually evil.
#112 Old 20th Jul 2009 at 7:56 PM
Si. Glad I could be of some help, and that your debugging proved fruitful. I just HATE off by one errors and typos that are accepted by compilers / interpreters. They should know what we meant, not what we typed, damn it.

Please spay or neuter your pets. --- Cat Music Video! --- my meshes
Mad Poster
Original Poster
#113 Old 20th Jul 2009 at 9:09 PM
[Well, chuckleing, anyway. ] The trouble with those is that what we type is often valid code but not valid programming...

...and how would the computer KWIM, anyway? Any systematic correction of such errors is bound to miscorrect occasional nonerrors.

Your suggestion was very helpful indeed. Domo arrigato. Danke Shoen. Shoulda thunk of it meself. .... duh ....

Now that it`s workin`, I gotsta start seein` whats I kin DOES wi` it.

Quoted from the top of the source file:
# Possible funtions:
# File sanity checker
# CAS screen switcher: http://www.modthesims2.com/showthread.php?t=275562
# Package Watcher http://www.modthesims2.com/showthread.php?t=222060
# Neighborhood Backup/Restore
# Cosmetic Surgery
# Genetic Therapy
# Manager for multiple conflicting packages; automaticaly disable active packages when enabling conflicting replacement
# Collection Creation
# Replacement for HomeCrafter: Make walls, floors, paintings, etc.

Which of these ideas do you think would be EASIEST to get working, and what local resources can you point me to that would help with any of them? Feel free to add any ideas you might have, too.

In addition to being able to process the file structure itself [mandatory for any program of this sort], my program is currently able to READ all of the following:

* GLOB records,
* NREF records,
* All known variants of the STR# record [same format, different uses, different ID codes].

The EBCF3E27 record is read, but not processed into fields. With decompression working [and almost finished], I`ll be researching additional record formats, in search of [1] the ability to process more record types and [2] records with values that it might be useful or interesting to change.

Does anyone know where to find information about the record that stores the name of the household living on any given lot? It should be easy and useful to provide the ability to rename a household, something the game does not provide, and would be suitable for making the learning tool also be a useful tool.

Edit: Hmm, I suppose re-implimenting my own version of Hack Conflict Detection would be possibly useful, too, and I don`t even need to add any record formats to the program to get started on THAT!

This Space Intentionally Left Blank
Test Subject
#114 Old 1st Aug 2009 at 10:59 PM
If you are still looking for a name try: The Simodifier or Package2sims. Hope this works well and when it does/or if it does,please publish it. SimPE doesn't even run on my pc it always freezes when it is loading before the program fully loads! lol! This will be better... I hope!
Mad Poster
Original Poster
#115 Old 3rd Aug 2009 at 5:53 PM
Thank you, Coolcrash05. I like "Simodifier" better than anything else I`ve seen so far.

General update:

* The decompression is now fully finished, and seems to be working fine.
* The records that store the names of the households living on the various lots are a collection of STR# records in the neighborhood package. As far as I can tell, they are the ONLY STR# records in that package.
* Next on the agenda: building code to WRITE a [possibly modified] *.package file to disk. When I get that working, I plan on adding code to allow changing the family name displayed on the neighborhood screen.

This Space Intentionally Left Blank
Test Subject
#116 Old 19th May 2010 at 3:46 AM
@G.O.D.: I used homecrafter but the thing I made didn't fit or meant the regulations of the game. OH well..... guess it's back to maxis edits for me....... I hate my computer. It's an old Compaq pesario 7500 series. I think I am going to save up for a new one.

Melody Dragon of Darkheart
Wish I was a Better Creator!
Mad Poster
Original Poster
#117 Old 28th May 2010 at 5:24 PM
This project has been on the "back burner" for a while, but I`m using additional code that I`ve added under the "Simodifier" umbrella to manage my downloads. I have the duplicate-file checker, discussed earlier, if I recall correctly, that I use to prune duplicate files from my Downloads directory tree, and I have added a subpackage [Python-package, not Sims2-package] that scans my Installed Archives folder, my Uninstalled Archives folder [which contains my Installed Archives folder as a sub-folder], and my Sims 2 Downloads folder, and compares what it finds there by MTS file ID number, reporting when it finds anommolies[sp?], such as installed content whose archive is AWOL, and things that appear multiple times in the Archives folders. I have a suplimentary function there that retrieves a MTS download-link ["getfile.php"] from the clipboard, extracts the file ID number, and reports where in the Archives and Sims2 Downloads directory trees that ID was found. If I get a result of None from both searches, I know that I haven`t [re-]downloaded the item since I lost my Linux file-server and don`t have it installed in Downloads [although some items get installed elsewhere]. If I get a location back from the Archives section, I know that I have downloaded the file since then, and if I get a location back from the Sims 2 Downloads section, I know it is installed there. I still have a LOT of items in the report on installed content with missing archives. Attempting to replace those missing archives is taking a lot of my attention, and is part of why this project has taken back-burner status. I am also discovering a LOT of stuff that I`ve Thanked [means I`ve downloaded it] that shows up in neither the Archives nor Downloads sections, meaning that I`ve lost the archive, but hadn`t yet installed it. ::sigh:: With Sims 2, if it isn`t one thing, it`s another. ::shrug::

This Space Intentionally Left Blank
Page 5 of 5
Back to top