extra TXTR remover - now with GUI - 20090301
Flag Download
Posts: 137
Thanks: 3822 in 12 Posts
http://www.moreawesomethanyou.com/s...pic,8279.0.html
GPL'd source, and useful discussion there.
Posts: 5,358
Thanks: 111973 in 267 Posts
Posts: 5,358
Thanks: 111973 in 267 Posts
done
- read in a package file
- decompress compressed stuff (thank you, benrq, for decompress)
- find the TXMTs and TXTRs
- decode the TXMTs
- figure out which TXMTs use which TXTRs
to do
- figure out which TXTRs to nuke and which to keep
- create an updated package header that reflects the changed number of resources
- create *correct* byte arrays for updated TXMTs with changed texture reference strings (string length likely changed, so big chunks will need to shift)
- recompress stuff (just use benrq's compress, no problem)
- create a new DIR that reflects the changed number of resources
- write out the new, updated, shrunken, recompressed package file, nuke the old one
Not terribly complex, just needs lots of attention to detail, very easy to screw this up with just a small mistake somewhere. Ah, fun with low level byte stream io. :D
Huge hugs for everyone who wrote the DBPF format wiki, and of course, benrq over at MATY for his port of zlib compress / decompress functions.
Posts: 3,129
Thanks: 4628 in 9 Posts
3) I don't plan to do a byte by byte comparison of TXTR data to see its a duplicate. |
Grant me the serenity to accept the things I cannot change, the courage to change the things I cannot accept, and the wisdom to hide the bodies of those I had to kill because they pissed me off.
Posts: 5,358
Thanks: 111973 in 267 Posts
hair with headband or flower or whatever
- N hair groups all use texture A
- headband/flower/whatever uses texture B
alpha skirt (not fun to remove dupes for 30 recolors by hand...)
- skirt inside and outside both use texture A
- legs use texture B
We want just one copy of texture A, but without nuking texture B, so I have to check which TXTRs match and which don't. And of course, black hair with elder grey, but there, the TXMT names tell us which TXMT is elder, so its TXTR references are grey, and non-elder are black.
Posts: 547
Thanks: 17015 in 40 Posts
Now, will this utility search for duplicates only within a single package, or will it be possible to search and kill duplicates across multiple packages so that all recolors of a hair can, for example, draw a veil or headband texture from a single recolor in a separate package?
Posts: 5,358
Thanks: 111973 in 267 Posts
benrq's compression is the same scheme used in jfade's Compressorizor, yes? |
Now, will this utility search for duplicates only within a single package, or will it be possible to search and kill duplicates across multiple packages so that all recolors of a hair can, for example, draw a veil or headband texture from a single recolor in a separate package? |
Multiple package scans would be nice for hair, and also for clothing. Lots of stuff I make uses the same bump map. I really should go back and make my old packages use the bump map in just one package, but I'm too lazy, so I'm writing this app to do it. Of course, there's a tradeoff there. You use less disk space, but you have to make sure you don't delete the one file that has the bump map for the 30 other files. Hm. Maybe I could stick the bump map in the mesh package? Could recolor files access it there?
Posts: 5,358
Thanks: 111973 in 267 Posts
CASE A: recolor with duplicate textures that REALLY ARE duplicates,
byte for byte, if you did a diff on the exported image files, it would say they're the same
I've seen Case A frequently, but mostly in clothing recolors. Does it occur often in hair? The duplicate texture remover I've been writing handles Case A just fine, removes extra TXTRs, updates TXMTs appropriately, will write out the new file (once I finish file writing).
CASE B: recolor with several textures that look like duplicates at first glance, but aren't
Upon extremely close inspection, they're jittered by one or two pixels so an automated comparison considers them different, even though for all intents and purposes, they're the same texture
I've seen Case B in some custom hair recolors I downloaded, but they were all by one creator, so perhaps its just that one person? In case B, my app thinks there are no duplicates. So I would have to write a UI that displays all the textures and lets the user select which ones should be treated as duplicates. That would be a pain in the butt. I need to know if its worthwhile or not. If its an infrequent case, people can just fix those files manually with SimPE. If its common, I would try to make the process less manual and tedious.
Posts: 5,358
Thanks: 111973 in 267 Posts
======================================================
Progress Report - Feb 17, 2009
***************************************
Alpha Software Now Available - TESTERS WANTED
***************************************
The alpha version of the extraTXTRremove program is now available for testing.
Please note that the alpha version is not the final, full featured version.
It is not meant for the general public yet, but rather for testers.
alpha version features
- Windows only
- command line interface
- new file is named "oldfilename_NO_DUPES.package" by default,
does NOT replace old file, unless you specifically tell it to
- works on a single package file at a time, not directories or multiple files
planned full version features (not yet done)
- Windows and Mac
- graphical user interface
- new file always replaces old file
- work on all files in a given directory, individually...
- ... or compare textures in several package files
- set TXMTs in several packages to use TXTR in one package
- optional ability to tell the program which textures should be treated as duplicates, for those special case hair recolors where the textures look identical, but aren't quite
Installation, Usage, Testing
- Download both the extraTXTRremove application and the Microsoft C Runtime Libraries.
- Unzip both *.zip files, put the contents in the same folder.
- To run the application, type the following at the command prompt:
./extraTXTRremove.exe FILENAME.package
If there are duplicates, the app will create a new, no dupes file with "_NO_DUPES" at the end of the filename, before ".package". Or type this and make both filenames the same if you want to overwrite the old file.
./extraTXTRremove.exe FILENAME_OLD.package FILENAME_NEW.package
The program will spew lots of debugging text at you, and then report either that there were no duplicate textures or create a new file without duplicate textures.
When testing, please let me know the following:
- Did my program create valid packages that work in game?
- If you look at the package with SimPE, did my program remove all the extra textures?
It should have, but you might see some that aren't exact duplicates.
Let me know if this is the case, ok?
- If something went wrong or didn't work, please give me the following info:
--- save the text my program printed to a text file
--- link or attachment with the recolor package file(s) you tested on
--- description of what went wrong, especially any error messages from the program, or Windows, the game, BodyShop, etc.
I don't expect this to harm your game in any way, but just to be safe,
test in an AnyGame, back up your files, blah blah standard disclaimer.
If anyone feels like testing this, thanks in advance.
*********************
DOWNLOAD - extraTXTRremove application (debug) (Feb 17 version)
DOWNLOAD - Microsoft C Runtime Libraries (debug)
*********************
.
Posts: 547
Thanks: 17015 in 40 Posts
Posts: 5,358
Thanks: 111973 in 267 Posts
Posts: 547
Thanks: 17015 in 40 Posts
Amber - Do you have .NET? I'll probably need to give you MicrsoSoft redist dll's so the app will run. Stupid MicroSoft. Why can't the OS just ship with all the dll's users will need to run apps built with MicroSoft IDEs? 'Cause that would be easy and make sense, that's why. Grumble grumble... |
I do have .NET, yes. Toss me a link and I can download those dll's.
Posts: 5,358
Thanks: 111973 in 267 Posts
Posts: 5,358
Thanks: 111973 in 267 Posts
download in post #10
Change from the Feb 16 version. Instead of the output file always being named "foo.package", you can now specify an output file name. It can rewrite the old input file if you want. Or if you don't specify an output filename, the output file will be named oldFileName_NO_DUPES.package.
-----------------------
To run the application, type the following at the command prompt:
./extraTXTRremove.exe FILENAME.package
If there are duplicates, the app will create a new, no dupes file with "_NO_DUPES" at the end of the filename, before ".package". Or type this and make both filenames the same if you want to overwrite the old file.
./extraTXTRremove.exe FILENAME_OLD.package FILENAME_NEW.package
Posts: 547
Thanks: 17015 in 40 Posts
Posts: 5,358
Thanks: 111973 in 267 Posts
I'm working on the GUI and the ability to check for duplicates across multiple files. Will also continue to test on any and all duplicate txtr files I can find. This will probably take me a week or so...
Posts: 547
Thanks: 17015 in 40 Posts
Posts: 5,358
Thanks: 111973 in 267 Posts
New version, added a graphical user interface, see post #1.
Features not yet available in the 20090301 version, but planned for the future...
- Mac version
- look for duplicate across files, not just within individual files
- let users choose "nearly identical" textures, and treat them as duplicates
Posts: 59
Thanks: 67 in 1 Posts
Unfortunately, I haven't seen much of a difference in file size. Only a couple of my files were reported as having duplicate textures, and I only lost a few kb after duplicate removal. Either my files are cleaner than I thought or perhaps the program isn't as ruthless as I thought it would be.
Hopefully when you implement user-chosen "nearly identical" texture removal I'll see more of a file size difference. (Also, all my downloads are Compressorized if that has some effect.)
Thank you again for your work on this program. The GUI is very intuitive and the program itself works very quickly without any problems.
Forgot to mention: I like the Favorites feature. It makes navigating to my folders easier. Also, I was able to select multiple files at once without SHIFT or CTRL-clicking, but simply by clicking on the first file I wanted and dragging down. It would be nice if whole directories could be selected at once, rather than having to open each directory, select files, move up, open another directory, select files, etc.
Posts: 5,358
Thanks: 111973 in 267 Posts
This program is not intended for most files. Most files do not have duplicate textures. You're only likely to see them if the file is a hair recolor, or a clothing recolor of a mesh with multiple mesh groups that use the same texture. The program has two main uses
- for creators, automatic dupe TXTR removal when they make hair or clothing with multiple mesh groups
- for downloaders, when they download hair / clothing from a creator that didn't bother to remove dupes, or didn't know how, whatever
Thanks for reporting "no errors", that's really good to hear. If you ever do get an error, feel free to report that too.
Glad the GUI makes sense. Made it similar to jfade's Compressorizer, since that makes lots of sense to me. The "favorites" feature is part of the FLTK gui library, so can't take credit for that, but glad you like it.
I agree that being able to select a directory (and all subdirectories too) would be quite handy. I'll try to add that in a future version.
Posts: 59
Thanks: 67 in 1 Posts
I loaded up all the files I ran through the program in BodyShop and they're working fine. I will test them in game tomorrow. (UPDATE: Files are working in game, too.)
Posts: 5,358
Thanks: 111973 in 267 Posts
I forgot to say, I plan to add "pick near identical textures" to remove because I have run into these in hair files. (sometimes this is deliberate and necessary, sometimes it just wastes space)
Just FYI, do NOT run this program on "mixed method" default skin files. I don't think you planned to, but just in case, don't. Such files come in pairs where one file has face textures, but no TXMTs. If it has duplicates, they might be removed, but the TXMTs in the OTHER FILE will not be changed, so the skin will break. I'll address this in a future version. I know there's a Maxis issue with face TXTRs and skins, so it may be that my program should just never be used on default skins.
Posts: 59
Thanks: 67 in 1 Posts
I forgot to say, I plan to add "pick near identical textures" to remove because I have run into these in hair files. (sometimes this is deliberate and necessary, sometimes it just wastes space) |
I wonder, will this also pick up on unused textures in hair files?
I use the Wardrobe Wrangler to remove age groups like toddler, child, etc. that use Maxis hairs since the custom hair mesh doesn't have the appropriate resources for those ages. The thing is, I believe the WW doesn't actually delete the Maxis textures from the package files (since you can still see them in the texture preview window) but just makes it so they don't show up in the catalog.
Using your user-preview-select feature, will we be able to view those Maxis textures and delete them from there?
Ah, I wish I knew about the skin thing last night. As it happens, I did exactly that. I have a default skinset blend I made using the mixed method. I ran it through the program, but it didn't find any duplicate textures. (Zero textures were removed, old file size and new file sizes are the same.) I am using the default skins right now and they appear to still be working. I will replace the files with my backups, though, since this is unsafe.
Posts: 5,358
Thanks: 111973 in 267 Posts
The reason I'm concerned about default skins is I have seen mixed method packages that used the same face for male and female, or for two or more ages. The TXMTs for those face textures were in a different file, and since I don't yet compare across files, such a skin could get messed up.
I didn't know hair could have unused textures. I'll look into removing that, but don't expect that anytime soon. It might end up being a separate program that does other stuff with hair packages to clean them up.
Page 1 of 2
|