Discussion:
Migrating from Qmail Rocks install on Solaris to Qmail Toaster on CentOS
Casey
2011-09-26 21:45:28 UTC
Permalink
I'm trying to migrate all of the domains, user accounts, and mail from
an old Sun Ultra2 running Solaris 9 with vpopmail 5.4.10 and mysql Ver.
14.7 distribution 4.1.21 to a newer server running CentOS 5 with Qmail
Toaster.

There are a couple of issues I'm running into - the current server is
setup with vpopmail install on its own set of disk arrays (/u1), and
then as that drive ran out of room, /u2 and /u3 were added, domains were
moved to those partitions and symlinks created to point to /u1

The other problem is that vpopmail is compiled with "many domains", "big
user dirs", and "learn passwords". The version of vpopmail on the new
server (5.4.17) is compiled without those features, which can easily be
resolved by recompiling, which I've done but I really want to
standardize things so that I don't have to recompile every time I want
to update.

So I'm looking for some help migrating everything to the new box. There
are a little over 300 domains, a few thousand users, and about 160GB of
mail. I tried recompiling vpopmail on the new server, editing the
database dump file before importing it (did a search for /u1 and
replaced it with /home/vpopmail), did the same for the
/var/qmail/users/assign file, ran qmail-newu and then restarted the
qmail services. I was able to get things partially working, in the sense
that running vuserinfo would give me the users associated with the
domain, and show credentials and such, but vqadmin was screwed up and
would say "no postmaster for this domain" when I tried managing things
from vqadmin, and then when you click show users, it would say domain
does not exist.

Please help!

--
Casey

Smile Global Technical Support
Submit or check trouble tickets http://billing.smileglobal.com
www.smileglobal.com <http://www.smileglobal.com>
Jake vickers
2011-09-26 22:04:38 UTC
Permalink
On 09/26/2011 05:45 PM, Casey wrote:
> I'm trying to migrate all of the domains, user accounts, and mail from
> an old Sun Ultra2 running Solaris 9 with vpopmail 5.4.10 and mysql
> Ver. 14.7 distribution 4.1.21 to a newer server running CentOS 5 with
> Qmail Toaster.
>
> There are a couple of issues I'm running into - the current server is
> setup with vpopmail install on its own set of disk arrays (/u1), and
> then as that drive ran out of room, /u2 and /u3 were added, domains
> were moved to those partitions and symlinks created to point to /u1
>
> The other problem is that vpopmail is compiled with "many domains",
> "big user dirs", and "learn passwords". The version of vpopmail on the
> new server (5.4.17) is compiled without those features, which can
> easily be resolved by recompiling, which I've done but I really want
> to standardize things so that I don't have to recompile every time I
> want to update.

This sounds like a rather manual process, unless someone else has done
something similar and can offer insight/scripts. From what I'm thinking
(shooting from the hip):
You may try a dump of the database (or vuserinfo scripted) to get the
usernames (emails) and passwords. This in a text file can be scripted to
run in a loop to create the users on the new server.
As far as the emails - rsync them over to the new server. Assuming you
were using courier, the dir structure should remain the same.


---------------------------------------------------------------------------------
Qmailtoaster is sponsored by Vickers Consulting Group (www.vickersconsulting.com)
Vickers Consulting Group offers Qmailtoaster support and installations.
If you need professional help with your setup, contact them today!
Casey
2011-09-26 23:19:38 UTC
Permalink
Thanks Jake. I'm hoping to make this process as automated as possible...

What I'm wondering, is does the database need to be converted somehow to
work properly on the new server since the old server is running mysql 4
and the new one is mysql 5? Also, in theory, if I recompiled vpopmail to
include the same features as the old server is running, and then
modified the sql dump file to replace the domain paths from the old
server with the correct ones on the new server (/home/vpopmail/domains),
and then rsync'd the mail over and kept it in the same type of structure
as it was on the old server...shouldn't that work?

When I first tried doing that, I had copied over all of the config
files, hadn't copied all of the data from the domain folders, and hadn't
done any compiling beyond enabling many domains. Like I mentioned
before, I was able to get vuserinfo to recognize the accounts with the
quotas, username, password, encrypted password, etc...but the major
problem was that vqadmin would show the domain, but wouldn't recognize
that there were accounts associated with it.

I guess I can kind of break this project up into to parts...my main
concern is getting everything moved to the new server, and then once
thats done I'd like to standardize things so that in the future if I
want to migrate I can just use the backup/restore scripts. I'm not sure
if this is better to try to tackle all at once, or two break up into
multiple tasks. It just seems like theres gotta be a way that this can
be done without a ton of scripting and manual entry - especially
considering i have everything in the database, I just need to make sure
that the domains/accounts are mapped to the correct location in the file
system.

Another question...theres been alot of talk on here about enabling "many
domains" vs. disabling it. As far as I know, in the current version it
is disabled, so when you create a new domain it creates a table for that
domain, opposed to putting them all under the vpopmail table. Any idea
how this will be setup in the future? Our existing system has it
enabled, and we have 300+ domains with alot of users, so naturally it
seems easier to try to keep things the same on our migration, but that
also means recompiling vpopmail every time we want to update, so are we
better off trying to convert our "many domains" database to the current
format?

Thanks in advance,
Casey

Smile Global Technical Support
Submit or check trouble tickets http://billing.smileglobal.com
www.smileglobal.com <http://www.smileglobal.com>

On 9/26/11 3:04 PM, Jake vickers wrote:
> On 09/26/2011 05:45 PM, Casey wrote:
>> I'm trying to migrate all of the domains, user accounts, and mail
>> from an old Sun Ultra2 running Solaris 9 with vpopmail 5.4.10 and
>> mysql Ver. 14.7 distribution 4.1.21 to a newer server running CentOS
>> 5 with Qmail Toaster.
>>
>> There are a couple of issues I'm running into - the current server is
>> setup with vpopmail install on its own set of disk arrays (/u1), and
>> then as that drive ran out of room, /u2 and /u3 were added, domains
>> were moved to those partitions and symlinks created to point to /u1
>>
>> The other problem is that vpopmail is compiled with "many domains",
>> "big user dirs", and "learn passwords". The version of vpopmail on
>> the new server (5.4.17) is compiled without those features, which can
>> easily be resolved by recompiling, which I've done but I really want
>> to standardize things so that I don't have to recompile every time I
>> want to update.
>
> This sounds like a rather manual process, unless someone else has done
> something similar and can offer insight/scripts. From what I'm
> thinking (shooting from the hip):
> You may try a dump of the database (or vuserinfo scripted) to get the
> usernames (emails) and passwords. This in a text file can be scripted
> to run in a loop to create the users on the new server.
> As far as the emails - rsync them over to the new server. Assuming you
> were using courier, the dir structure should remain the same.
>
>
> ---------------------------------------------------------------------------------
>
> Qmailtoaster is sponsored by Vickers Consulting Group
> (www.vickersconsulting.com)
> Vickers Consulting Group offers Qmailtoaster support and
> installations.
> If you need professional help with your setup, contact them today!
> ---------------------------------------------------------------------------------
>
> Please visit qmailtoaster.com for the latest news, updates, and
> packages.
> To unsubscribe, e-mail:
> qmailtoaster-list-***@qmailtoaster.com
> For additional commands, e-mail:
> qmailtoaster-list-***@qmailtoaster.com
>
>
Jake vickers
2011-09-27 00:21:42 UTC
Permalink
On 09/26/2011 07:19 PM, Casey wrote:
> Thanks Jake. I'm hoping to make this process as automated as possible...
>
> What I'm wondering, is does the database need to be converted somehow
> to work properly on the new server since the old server is running
> mysql 4 and the new one is mysql 5? Also, in theory, if I recompiled
> vpopmail to include the same features as the old server is running,
> and then modified the sql dump file to replace the domain paths from
> the old server with the correct ones on the new server
> (/home/vpopmail/domains), and then rsync'd the mail over and kept it
> in the same type of structure as it was on the old server...shouldn't
> that work?
>
> When I first tried doing that, I had copied over all of the config
> files, hadn't copied all of the data from the domain folders, and
> hadn't done any compiling beyond enabling many domains. Like I
> mentioned before, I was able to get vuserinfo to recognize the
> accounts with the quotas, username, password, encrypted password,
> etc...but the major problem was that vqadmin would show the domain,
> but wouldn't recognize that there were accounts associated with it.

Sure, you can do it this way, but you'll be doing the work twice. Once
to modify the stock packages now to ease the migration, and then again
to standardize your environment to this project. IMHO it would be time
best spent to do standardize it now, and forget worrying about
standardizing it later.


>
> I guess I can kind of break this project up into to parts...my main
> concern is getting everything moved to the new server, and then once
> thats done I'd like to standardize things so that in the future if I
> want to migrate I can just use the backup/restore scripts. I'm not
> sure if this is better to try to tackle all at once, or two break up
> into multiple tasks. It just seems like theres gotta be a way that
> this can be done without a ton of scripting and manual entry -
> especially considering i have everything in the database, I just need
> to make sure that the domains/accounts are mapped to the correct
> location in the file system.

So if you wrote a script on the source box to do something like "for
each vuserinfo" and walk the directory containing the users to get a
text output of the username (email address) and password, the ran a
script to "vadduser" each line in that text output you would have all of
the users recreated on the destination server. Then rsync each user's
dir to the new location (ie: rsync -avz --progress -e ssh /u/domain.com/
***@destination:/home/vpopmail/domain/) and you would have the user on
the new machine, and a copy of their mailstore on the destination server
as well. Maybe keep a looped script running to rsync the mailstore using
the "--delete" option to keep a sync'ed copy of the mailstore on the
destination until you have had a chance to test it out and feel
comfortable changing DNS over.
Then you don't have to futz with DB structures, translations, changes,
etc. And you have the added benefit of moving 1 domain at a time if you
want, so you can make the transition in a staged method.





>
> Another question...theres been alot of talk on here about enabling
> "many domains" vs. disabling it. As far as I know, in the current
> version it is disabled, so when you create a new domain it creates a
> table for that domain, opposed to putting them all under the vpopmail
> table. Any idea how this will be setup in the future? Our existing
> system has it enabled, and we have 300+ domains with alot of users, so
> naturally it seems easier to try to keep things the same on our
> migration, but that also means recompiling vpopmail every time we want
> to update, so are we better off trying to convert our "many domains"
> database to the current format?
>

So I have not seen any performance gain either way. The only benefit
thus far has been to ease writing of programs in using one table vs
many. I personally have a machine that has 450 domains, 4200 users, and
processes somewhere in the ballpark of 2 million emails a month (inbound
and outbound). This is on a P4 3Ghz with 2G of RAM. Most are POP3, and
I've had to tweak that to keep from hitting the maximum number of
connections, plus a few other tweaks, but nothing MAJOR.
If we do make a DB change in the future (and we probably will, since a
lot of people ask for it and there is no negative impact of changing) it
will be for a different version of QMT. If, and this is probably not
very likely, we change the DB format for the 1.3 branch, we will have
conversion scripts in place to assist with migrating beforehand.
Casey
2011-09-27 18:56:02 UTC
Permalink
How exactly would I go about writing the script? I'm not the most
proficient with shell scripting, but I can poke my way around. If I had
a little more to start with, then I could probably get the jist of it.

Would it be possible to extract other details of each account
(directory, comment, quota, usage, last auth, last ip), and have them
inputted when the account is created? I'm just trying to figure out the
easiest way to go about things. Ultimately, I will need to get all of
that info added to each user, and it just makes the most sense to do it
all at once when I first recreate the accounts.


Casey

Smile Global Technical Support
Submit or check trouble tickets http://billing.smileglobal.com
www.smileglobal.com <http://www.smileglobal.com>

On 9/26/11 5:21 PM, Jake vickers wrote:
> On 09/26/2011 07:19 PM, Casey wrote:
>> Thanks Jake. I'm hoping to make this process as automated as possible...
>>
>> What I'm wondering, is does the database need to be converted somehow
>> to work properly on the new server since the old server is running
>> mysql 4 and the new one is mysql 5? Also, in theory, if I recompiled
>> vpopmail to include the same features as the old server is running,
>> and then modified the sql dump file to replace the domain paths from
>> the old server with the correct ones on the new server
>> (/home/vpopmail/domains), and then rsync'd the mail over and kept it
>> in the same type of structure as it was on the old server...shouldn't
>> that work?
>>
>> When I first tried doing that, I had copied over all of the config
>> files, hadn't copied all of the data from the domain folders, and
>> hadn't done any compiling beyond enabling many domains. Like I
>> mentioned before, I was able to get vuserinfo to recognize the
>> accounts with the quotas, username, password, encrypted password,
>> etc...but the major problem was that vqadmin would show the domain,
>> but wouldn't recognize that there were accounts associated with it.
>
> Sure, you can do it this way, but you'll be doing the work twice. Once
> to modify the stock packages now to ease the migration, and then again
> to standardize your environment to this project. IMHO it would be time
> best spent to do standardize it now, and forget worrying about
> standardizing it later.
>
>
>>
>> I guess I can kind of break this project up into to parts...my main
>> concern is getting everything moved to the new server, and then once
>> thats done I'd like to standardize things so that in the future if I
>> want to migrate I can just use the backup/restore scripts. I'm not
>> sure if this is better to try to tackle all at once, or two break up
>> into multiple tasks. It just seems like theres gotta be a way that
>> this can be done without a ton of scripting and manual entry -
>> especially considering i have everything in the database, I just need
>> to make sure that the domains/accounts are mapped to the correct
>> location in the file system.
>
> So if you wrote a script on the source box to do something like "for
> each vuserinfo" and walk the directory containing the users to get a
> text output of the username (email address) and password, the ran a
> script to "vadduser" each line in that text output you would have all
> of the users recreated on the destination server. Then rsync each
> user's dir to the new location (ie: rsync -avz --progress -e ssh
> /u/domain.com/ ***@destination:/home/vpopmail/domain/) and you would
> have the user on the new machine, and a copy of their mailstore on the
> destination server as well. Maybe keep a looped script running to
> rsync the mailstore using the "--delete" option to keep a sync'ed copy
> of the mailstore on the destination until you have had a chance to
> test it out and feel comfortable changing DNS over.
> Then you don't have to futz with DB structures, translations, changes,
> etc. And you have the added benefit of moving 1 domain at a time if
> you want, so you can make the transition in a staged method.
>
>
>
>
>
>>
>> Another question...theres been alot of talk on here about enabling
>> "many domains" vs. disabling it. As far as I know, in the current
>> version it is disabled, so when you create a new domain it creates a
>> table for that domain, opposed to putting them all under the vpopmail
>> table. Any idea how this will be setup in the future? Our existing
>> system has it enabled, and we have 300+ domains with alot of users,
>> so naturally it seems easier to try to keep things the same on our
>> migration, but that also means recompiling vpopmail every time we
>> want to update, so are we better off trying to convert our "many
>> domains" database to the current format?
>>
>
> So I have not seen any performance gain either way. The only benefit
> thus far has been to ease writing of programs in using one table vs
> many. I personally have a machine that has 450 domains, 4200 users,
> and processes somewhere in the ballpark of 2 million emails a month
> (inbound and outbound). This is on a P4 3Ghz with 2G of RAM. Most are
> POP3, and I've had to tweak that to keep from hitting the maximum
> number of connections, plus a few other tweaks, but nothing MAJOR.
> If we do make a DB change in the future (and we probably will, since a
> lot of people ask for it and there is no negative impact of changing)
> it will be for a different version of QMT. If, and this is probably
> not very likely, we change the DB format for the 1.3 branch, we will
> have conversion scripts in place to assist with migrating beforehand.
>
Casey
2011-09-29 08:29:44 UTC
Permalink
Well...I was able to get the migrate-domain script i found working,
turns out the reason it wouldn't run, was because the Solaris server was
setup with tcsh as the default shell, instead of bash. I know...simple,
right?

Everything works pretty well so far. I run the script on each domain, it
creates a tarball of each mailbox, and generates a script that creates
the domain, each user, sets the password, quota, etc. It also recreates
the mailing lists. I had a few issues, because of the different paths
being used between the servers, but I wrote a simple script that
re-created the symlinks for each mailing list, and modified each file to
include the correct domain path for the mailing list.

Here is the next thing I'm not sure about...what is the best way for me
to re-create all of the user aliases? Looks like they are all stored in
the database on the old server, but is there an easy way that I can
"import" them?

Another issue I noticed, is that when I run vdominfo, the total user
count is only shown for the first domain I added - the other domains
simple say "0". However, vqadmin and vuserinfo both show all of the
domains, users, and appear to be working fine.

Thanks,

Casey

Smile Global Technical Support
Submit or check trouble tickets http://billing.smileglobal.com
www.smileglobal.com <http://www.smileglobal.com>

On 9/26/11 5:21 PM, Jake vickers wrote:
> On 09/26/2011 07:19 PM, Casey wrote:
>> Thanks Jake. I'm hoping to make this process as automated as possible...
>>
>> What I'm wondering, is does the database need to be converted somehow
>> to work properly on the new server since the old server is running
>> mysql 4 and the new one is mysql 5? Also, in theory, if I recompiled
>> vpopmail to include the same features as the old server is running,
>> and then modified the sql dump file to replace the domain paths from
>> the old server with the correct ones on the new server
>> (/home/vpopmail/domains), and then rsync'd the mail over and kept it
>> in the same type of structure as it was on the old server...shouldn't
>> that work?
>>
>> When I first tried doing that, I had copied over all of the config
>> files, hadn't copied all of the data from the domain folders, and
>> hadn't done any compiling beyond enabling many domains. Like I
>> mentioned before, I was able to get vuserinfo to recognize the
>> accounts with the quotas, username, password, encrypted password,
>> etc...but the major problem was that vqadmin would show the domain,
>> but wouldn't recognize that there were accounts associated with it.
>
> Sure, you can do it this way, but you'll be doing the work twice. Once
> to modify the stock packages now to ease the migration, and then again
> to standardize your environment to this project. IMHO it would be time
> best spent to do standardize it now, and forget worrying about
> standardizing it later.
>
>
>>
>> I guess I can kind of break this project up into to parts...my main
>> concern is getting everything moved to the new server, and then once
>> thats done I'd like to standardize things so that in the future if I
>> want to migrate I can just use the backup/restore scripts. I'm not
>> sure if this is better to try to tackle all at once, or two break up
>> into multiple tasks. It just seems like theres gotta be a way that
>> this can be done without a ton of scripting and manual entry -
>> especially considering i have everything in the database, I just need
>> to make sure that the domains/accounts are mapped to the correct
>> location in the file system.
>
> So if you wrote a script on the source box to do something like "for
> each vuserinfo" and walk the directory containing the users to get a
> text output of the username (email address) and password, the ran a
> script to "vadduser" each line in that text output you would have all
> of the users recreated on the destination server. Then rsync each
> user's dir to the new location (ie: rsync -avz --progress -e ssh
> /u/domain.com/ ***@destination:/home/vpopmail/domain/) and you would
> have the user on the new machine, and a copy of their mailstore on the
> destination server as well. Maybe keep a looped script running to
> rsync the mailstore using the "--delete" option to keep a sync'ed copy
> of the mailstore on the destination until you have had a chance to
> test it out and feel comfortable changing DNS over.
> Then you don't have to futz with DB structures, translations, changes,
> etc. And you have the added benefit of moving 1 domain at a time if
> you want, so you can make the transition in a staged method.
>
>
>
>
>
>>
>> Another question...theres been alot of talk on here about enabling
>> "many domains" vs. disabling it. As far as I know, in the current
>> version it is disabled, so when you create a new domain it creates a
>> table for that domain, opposed to putting them all under the vpopmail
>> table. Any idea how this will be setup in the future? Our existing
>> system has it enabled, and we have 300+ domains with alot of users,
>> so naturally it seems easier to try to keep things the same on our
>> migration, but that also means recompiling vpopmail every time we
>> want to update, so are we better off trying to convert our "many
>> domains" database to the current format?
>>
>
> So I have not seen any performance gain either way. The only benefit
> thus far has been to ease writing of programs in using one table vs
> many. I personally have a machine that has 450 domains, 4200 users,
> and processes somewhere in the ballpark of 2 million emails a month
> (inbound and outbound). This is on a P4 3Ghz with 2G of RAM. Most are
> POP3, and I've had to tweak that to keep from hitting the maximum
> number of connections, plus a few other tweaks, but nothing MAJOR.
> If we do make a DB change in the future (and we probably will, since a
> lot of people ask for it and there is no negative impact of changing)
> it will be for a different version of QMT. If, and this is probably
> not very likely, we change the DB format for the 1.3 branch, we will
> have conversion scripts in place to assist with migrating beforehand.
>
Pak Ogah
2011-09-30 07:44:56 UTC
Permalink
On 09/29/11 15:29, Casey wrote:
>
> Here is the next thing I'm not sure about...what is the best way for
> me to re-create all of the user aliases? Looks like they are all
> stored in the database on the old server, but is there an easy way
> that I can "import" them?
>

you can easily export import vpopmail mysql database using my previous
howto,
or just using heidisql by connecting to both server
Casey
2011-10-01 03:42:28 UTC
Permalink
Thanks Pak. My initial thought was to try to export the valias table
from the vpopmail database, and import it on the new server, however I
wasn't sure if that would work/be a good idea considering the fact that
I'm migrating the domains individually, because of the way the old
server's vpopmail is built, and the fact that vpopmail is installed on
/u1 on the old system, opposed to /home/vpopmail on the new one, and
last but not least, I wanted to play it safe.

What I ended up doing modifying the valias2dotqmail script I found on
the qmailtoaster wiki so that it created the .qmail-alias files in an
"aliases" folder in abc.com's root folder, and then I copied the
contents of that folder to the new server - wrote a script of my own
that used sed to search through the files and replace the /u1 path with
/home/vpopmail. This last step I haven't tested yet, but will be giving
it a shot tonight: using the dotqmail2valias program that is included
with vpopmail. It looks like it should work, so we'll see.

Oh and I solved my problem with the mailing lists...I realized that
after migrating a domain, all of the symlinks that were previously
created in the domain's root directory were pointing to the old path,
and it was going to be a hassle to have to manually recreate and edit
each one...so I wrote a script for that as well...I was actually quite
impressed with myself, because my scripting skills are limited at best.

--------

#!/bin/bash

# Created by Casey James Price -- (09/28/2011)
# Smile Global -- POP Server Migration Project -- fix-links.sh
# This script fixes the mailing list symlinks that were created when the
domain was migrated
# Replaces '/u1' with '/home/vpopmail'
# Use 'find' in a for loop to locate the bad symlinks, and then readlink
to show the destinaton of the actual file.
# Pipe the output to sed and do a regexp search/replace. Store this
value as 'x', and call 'ln' to recreate
# the symlink using '$x' as the destination and '$a' as the link name.
Use sed again to replace the old domain path
# inside of the file itself. Lastly, 'chown -R vpopmail.vchkpw $a'

for a in `find /home/vpopmail/domains/smileglobal.com -type l`;
do
x=`readlink $a | sed 's/u1/home\/vpopmail/'`
ln -nsf $x $a
`sed -i 's/u1/home\/vpopmail/g' $x`
`chown -R vpopmail.vchkpw $a`
done

-------

Casey

Smile Global Technical Support
Submit or check trouble tickets http://billing.smileglobal.com
www.smileglobal.com <http://www.smileglobal.com>

On 9/30/11 12:44 AM, Pak Ogah wrote:
> On 09/29/11 15:29, Casey wrote:
>>
>> Here is the next thing I'm not sure about...what is the best way for
>> me to re-create all of the user aliases? Looks like they are all
>> stored in the database on the old server, but is there an easy way
>> that I can "import" them?
>>
>
> you can easily export import vpopmail mysql database using my previous
> howto,
> or just using heidisql by connecting to both server
Eric Shubert
2011-10-01 14:02:35 UTC
Permalink
Nice job, Casey.

I hope you will consider taking a little time to document your
experiences here:
http://wiki.qmailtoaster.com/index.php/Migrating_from_non-toaster_qmail
That page has little content, so feel to edit/modify what's there.

Thanks for sharing. :)

--
-Eric 'shubes'

On 09/30/2011 08:42 PM, Casey wrote:
> Thanks Pak. My initial thought was to try to export the valias table
> from the vpopmail database, and import it on the new server, however I
> wasn't sure if that would work/be a good idea considering the fact that
> I'm migrating the domains individually, because of the way the old
> server's vpopmail is built, and the fact that vpopmail is installed on
> /u1 on the old system, opposed to /home/vpopmail on the new one, and
> last but not least, I wanted to play it safe.
>
> What I ended up doing modifying the valias2dotqmail script I found on
> the qmailtoaster wiki so that it created the .qmail-alias files in an
> "aliases" folder in abc.com's root folder, and then I copied the
> contents of that folder to the new server - wrote a script of my own
> that used sed to search through the files and replace the /u1 path with
> /home/vpopmail. This last step I haven't tested yet, but will be giving
> it a shot tonight: using the dotqmail2valias program that is included
> with vpopmail. It looks like it should work, so we'll see.
>
> Oh and I solved my problem with the mailing lists...I realized that
> after migrating a domain, all of the symlinks that were previously
> created in the domain's root directory were pointing to the old path,
> and it was going to be a hassle to have to manually recreate and edit
> each one...so I wrote a script for that as well...I was actually quite
> impressed with myself, because my scripting skills are limited at best.
>
> --------
>
> #!/bin/bash
>
> # Created by Casey James Price -- (09/28/2011)
> # Smile Global -- POP Server Migration Project -- fix-links.sh
> # This script fixes the mailing list symlinks that were created when the
> domain was migrated
> # Replaces '/u1' with '/home/vpopmail'
> # Use 'find' in a for loop to locate the bad symlinks, and then readlink
> to show the destinaton of the actual file.
> # Pipe the output to sed and do a regexp search/replace. Store this
> value as 'x', and call 'ln' to recreate
> # the symlink using '$x' as the destination and '$a' as the link name.
> Use sed again to replace the old domain path
> # inside of the file itself. Lastly, 'chown -R vpopmail.vchkpw $a'
>
> for a in `find /home/vpopmail/domains/smileglobal.com -type l`;
> do
> x=`readlink $a | sed 's/u1/home\/vpopmail/'`
> ln -nsf $x $a
> `sed -i 's/u1/home\/vpopmail/g' $x`
> `chown -R vpopmail.vchkpw $a`
> done
>
> -------
>
> Casey
>
> Smile Global Technical Support
> Submit or check trouble tickets http://billing.smileglobal.com
> www.smileglobal.com <http://www.smileglobal.com>
>
> On 9/30/11 12:44 AM, Pak Ogah wrote:
>> On 09/29/11 15:29, Casey wrote:
>>>
>>> Here is the next thing I'm not sure about...what is the best way for
>>> me to re-create all of the user aliases? Looks like they are all
>>> stored in the database on the old server, but is there an easy way
>>> that I can "import" them?
>>>
>>
>> you can easily export import vpopmail mysql database using my previous
>> howto,
>> or just using heidisql by connecting to both server



---------------------------------------------------------------------------------
Qmailtoaster is sponsored by Vickers Consulting Group (www.vickersconsulting.com)
Vickers Consulting Group offers Qmailtoaster support and installations.
If you need professional help with your setup, contact them today!
Casey
2011-10-01 20:28:20 UTC
Permalink
Thanks Eric. I'd be more than happy to document the whole process after
I've gotten through everything. I know at first I had a tough time
figuring out where to start on this project, and if I could document my
experiences, maybe it could prevent a few headaches on the next guy that
wants to do such a migration ;-)

One thing I haven't been able to figure out, is why vdomifo isn't
providing correct information when running it with the "-t" flag. I get
all zeroes ("0") when I simply run vdominfo -t, except for the first
domain that I added, which shows an excessively high user count (tens of
thousands if not more - accounts on the system, when that domain in fact
only has 240 - 250 accounts). How does vdominfo calculate the total
users? This probably isn't a huge deal, but its definitely nice being
able to run one command and see how many accounts are associated with
each domain. Ideas?

Thanks,

Casey

Smile Global Technical Support
Submit or check trouble tickets http://billing.smileglobal.com
www.smileglobal.com <http://www.smileglobal.com>

On 10/1/11 7:02 AM, Eric Shubert wrote:
> Nice job, Casey.
>
> I hope you will consider taking a little time to document your
> experiences here:
> http://wiki.qmailtoaster.com/index.php/Migrating_from_non-toaster_qmail
> That page has little content, so feel to edit/modify what's there.
>
> Thanks for sharing. :)
>
Eric Shubert
2011-10-02 07:01:58 UTC
Permalink
Thanks for taking the time to put together some documentation, Casey.
We'll all look forward to seeing it.

In the present vpopmail-toaster-5.4.17-1.3.7, a couple pieces of the
data in the vpopmail database are not always quite right. You've
identified one of them, and quotas is another.

In the case of quotas, it is reportedly fixed in vpopmail-5.4.33, which
has not yet been packaged for QMT (a vpopmail-toaster-5.4.33-1.3.8
package). We're hopeful that vqadmin will also once again work correctly
with that version, which is rumored.

I'm not certain whether or not the particular error to which you refer
has been identified or fixed in versions since 5.4.17. It would be
inappropriate for the authors (the kind folks at inter7 who work on
vpopmail, namely Matt Brookings) to try to fix anything like this in
older versions, as his development time with vpopmail is limited. We'll
need to upgrade to 5.4.33 and see if the problem persists in that
version before reporting a bug, and getting it fixed.

I've already written a script that updates the database appropriately
for upgrading from 5.4.17 to 5.4.27, but don't know if there are any
database changes since that would require special processing for
upgrading. That's pretty much where the process has stalled.

So how would you like to help getting vpopmail-toaster-5.4.33-1.3.8 put
together?

--
-Eric 'shubes'

On 10/01/2011 01:28 PM, Casey wrote:
> Thanks Eric. I'd be more than happy to document the whole process after
> I've gotten through everything. I know at first I had a tough time
> figuring out where to start on this project, and if I could document my
> experiences, maybe it could prevent a few headaches on the next guy that
> wants to do such a migration ;-)
>
> One thing I haven't been able to figure out, is why vdomifo isn't
> providing correct information when running it with the "-t" flag. I get
> all zeroes ("0") when I simply run vdominfo -t, except for the first
> domain that I added, which shows an excessively high user count (tens of
> thousands if not more - accounts on the system, when that domain in fact
> only has 240 - 250 accounts). How does vdominfo calculate the total
> users? This probably isn't a huge deal, but its definitely nice being
> able to run one command and see how many accounts are associated with
> each domain. Ideas?
>
> Thanks,
>
> Casey
>
> Smile Global Technical Support
> Submit or check trouble tickets http://billing.smileglobal.com
> www.smileglobal.com <http://www.smileglobal.com>
>
> On 10/1/11 7:02 AM, Eric Shubert wrote:
>> Nice job, Casey.
>>
>> I hope you will consider taking a little time to document your
>> experiences here:
>> http://wiki.qmailtoaster.com/index.php/Migrating_from_non-toaster_qmail
>> That page has little content, so feel to edit/modify what's there.
>>
>> Thanks for sharing. :)
>>



---------------------------------------------------------------------------------
Qmailtoaster is sponsored by Vickers Consulting Group (www.vickersconsulting.com)
Vickers Consulting Group offers Qmailtoaster support and installations.
If you need professional help with your setup, contact them today!
Casey
2011-10-02 07:17:54 UTC
Permalink
Thanks Eric. Wasn't sure if I was doing something wrong that was causing
this problem, or if it was a bug. What type of issues are there with
quotas? I'm sure I'll run into them eventually, but if I knew what to
watch out for, it would definitely be helpful.

Totally understandable about the development of vpopmail...we're all
fairly busy people these days, so it can certainly be difficult at times
to dedicate enough time to any one thing ;-)

I'm actually kinda honored that you're asking if I'd like to help work
on the vpopmail-toaster package. I must admit I'm still very much a
beginner at all of this stuff with qmailtoaster, but I'm a quick learner
and look forward to getting my experience in. With that being said...I'd
love to help! Just let me know what you need done, and I'll do my best
to figure out how to get it accomplished.

Thanks,

Casey

Smile Global Technical Support
Submit or check trouble tickets http://billing.smileglobal.com
www.smileglobal.com <http://www.smileglobal.com>

On 10/2/11 12:01 AM, Eric Shubert wrote:
> Thanks for taking the time to put together some documentation, Casey.
> We'll all look forward to seeing it.
>
> In the present vpopmail-toaster-5.4.17-1.3.7, a couple pieces of the
> data in the vpopmail database are not always quite right. You've
> identified one of them, and quotas is another.
>
> In the case of quotas, it is reportedly fixed in vpopmail-5.4.33,
> which has not yet been packaged for QMT (a
> vpopmail-toaster-5.4.33-1.3.8 package). We're hopeful that vqadmin
> will also once again work correctly with that version, which is rumored.
>
> I'm not certain whether or not the particular error to which you refer
> has been identified or fixed in versions since 5.4.17. It would be
> inappropriate for the authors (the kind folks at inter7 who work on
> vpopmail, namely Matt Brookings) to try to fix anything like this in
> older versions, as his development time with vpopmail is limited.
> We'll need to upgrade to 5.4.33 and see if the problem persists in
> that version before reporting a bug, and getting it fixed.
>
> I've already written a script that updates the database appropriately
> for upgrading from 5.4.17 to 5.4.27, but don't know if there are any
> database changes since that would require special processing for
> upgrading. That's pretty much where the process has stalled.
>
> So how would you like to help getting vpopmail-toaster-5.4.33-1.3.8
> put together?
>
Pak Ogah
2011-10-03 03:04:51 UTC
Permalink
On 10/01/11 21:02, Eric Shubert wrote:
> Nice job, Casey.
>
> I hope you will consider taking a little time to document your
> experiences here:
> http://wiki.qmailtoaster.com/index.php/Migrating_from_non-toaster_qmail
> That page has little content, so feel to edit/modify what's there.
>
> Thanks for sharing. :)
>
Eric, I think the title shoud be "Migrating from Qmailrocks", as Casey
was using Qmailrocks.
Although he is using Solaris on his old server, but I think his
documentation should be able followed by other Qmailrocks users that use
Linux who also want to migrate to QMT.
as Solaris and Linux is command and path almost identical. And I think
there are many Qmailrocks users who want to migrate since qmailrocks
website/manual and support forum no longer active. (my friend is one of
them but the company buying him Windows 2008 svr + ms. Exchange)




---------------------------------------------------------------------------------
Qmailtoaster is sponsored by Vickers Consulting Group (www.vickersconsulting.com)
Vickers Consulting Group offers Qmailtoaster support and installations.
If you need professional help with your setup, contact them today!
Eric Shubert
2011-10-03 03:21:11 UTC
Permalink
On 10/02/2011 08:04 PM, Pak Ogah wrote:
> On 10/01/11 21:02, Eric Shubert wrote:
>> Nice job, Casey.
>>
>> I hope you will consider taking a little time to document your
>> experiences here:
>> http://wiki.qmailtoaster.com/index.php/Migrating_from_non-toaster_qmail
>> That page has little content, so feel to edit/modify what's there.
>>
>> Thanks for sharing. :)
>>
> Eric, I think the title shoud be "Migrating from Qmailrocks", as Casey
> was using Qmailrocks.
> Although he is using Solaris on his old server, but I think his
> documentation should be able followed by other Qmailrocks users that use
> Linux who also want to migrate to QMT.
> as Solaris and Linux is command and path almost identical. And I think
> there are many Qmailrocks users who want to migrate since qmailrocks
> website/manual and support forum no longer active. (my friend is one of
> them but the company buying him Windows 2008 svr + ms. Exchange)
>
>
>
>
> ---------------------------------------------------------------------------------

While QMR is a non-toaster_qmail setup (and thus fits the description),
it's pretty specific, and is even a predecessor of sorts to QMT.
A separate page for QMR is fine with me.

--
-Eric 'shubes'


---------------------------------------------------------------------------------
Qmailtoaster is sponsored by Vickers Consulting Group (www.vickersconsulting.com)
Vickers Consulting Group offers Qmailtoaster support and installations.
If you need professional help with your setup, contact them today!
Casey
2011-10-04 08:18:06 UTC
Permalink
Ok guys, here is my next challenge...really, the biggest problem I'm
foreseeing at the moment:

So here I am joyfully (as I curse at the computer under my breath every
time I find a new problem...) migrating all 4000+ users of the 392
different domains using the migrate-domain script (if you aren't
familiar, it basically creates a tarball of every mailbox belonging to
the domain, and then generates a script that when run will recreate the
domain, user accounts, mailing lists and forwards --with a minor amount
of fuss after the fact on the new system) --- so that is all good and
well, but here is where the problem comes in...I have 165GB of data to
migrate, and in the end I plan on taking down "Server A" and having
"Server B" completely take over, except every second after data has been
migrated over to "Server B", there is a pretty good chance that new mail
is coming in for one of the 392 different domains.

The problem is that as long as that old server is still up, I need to
worry about keeping the mail content consistent between the old server
and the new. Obviously I'd like to have the time between domain
migration and server deployment down to a minimum, but some of my
customers are quite large (600 or so email accounts, each of which will
need to be reconfigured on their end at minimum just to make sure they
are using smtp-auth, and probably changing the incoming/outgoing server
settings on a few systems that are statically set to use the server's IP
versus hostname.

Normally I'd probably opt to use rsync to keep the new server in-tune
with the old one, but I'm not sure that is exactly an option, because
the old system has vpopmail compiled with "many-domains" enabled, in
addition to "big user dirs", so once again I'm trying to interface with
a different folder structure -- for example, on the old system we have
domains that are broken up in the domain's directory by number (i.e.;
domains/0/abc.com, domains/1/bca.com, domains/cad.com, etc) as well as
users broken up by number in a similar fashion (i.e.;
domains/abc.com/0/postmaster, domains/1/bca.com/demo,
domains/cad.com/2/test, etc). Since I'm trying to standardize, I don'
want to have to compile those features in...but that makes it difficult
to use something such as rsync to run against a domain folder with the
flags set that will delete files from the destination that don't exist
on the host. So what do I do? I need a fairly automated way to keep the
mail synchronized between the servers before we take the old one down.
At the moment, there is enough manual intervention required in migrating
the domains, that it doesn't seem realistic to try to get all the data
migrated and current the day that we decide to do all of this.

I'm really really really hoping that someone out there has some ideas
and can help me out with this one. I'd like the least amount of service
disruption possible, and can't run the risk of missing or losing email
for my customers. I want to get this to work, because my next project is
to setup a replicated qmailtoaster, which should be much easier to work
with and can use rsync and/or unison to get things in check.

There has gotta be a few Toaster Master's out there that have the
perfect solution...all I need is one good idea ;-)

Thanks in advance,


Casey

Smile Global Technical Support
Submit or check trouble tickets http://billing.smileglobal.com
www.smileglobal.com <http://www.smileglobal.com>

On 10/2/11 8:21 PM, Eric Shubert wrote:
> On 10/02/2011 08:04 PM, Pak Ogah wrote:
>> On 10/01/11 21:02, Eric Shubert wrote:
>>> Nice job, Casey.
>>>
>>> I hope you will consider taking a little time to document your
>>> experiences here:
>>> http://wiki.qmailtoaster.com/index.php/Migrating_from_non-toaster_qmail
>>> That page has little content, so feel to edit/modify what's there.
>>>
>>> Thanks for sharing. :)
>>>
>> Eric, I think the title shoud be "Migrating from Qmailrocks", as Casey
>> was using Qmailrocks.
>> Although he is using Solaris on his old server, but I think his
>> documentation should be able followed by other Qmailrocks users that use
>> Linux who also want to migrate to QMT.
>> as Solaris and Linux is command and path almost identical. And I think
>> there are many Qmailrocks users who want to migrate since qmailrocks
>> website/manual and support forum no longer active. (my friend is one of
>> them but the company buying him Windows 2008 svr + ms. Exchange)
>>
>>
>>
>>
>> ---------------------------------------------------------------------------------
>>
>
> While QMR is a non-toaster_qmail setup (and thus fits the
> description), it's pretty specific, and is even a predecessor of sorts
> to QMT.
> A separate page for QMR is fine with me.
>
Eric Shubert
2011-10-04 14:51:05 UTC
Permalink
As I see it, you could use 2 different processes (or bridges) for
migration. One process migrates the accounts, and is done only one time.
The 2nd process migrates the mail, and is done several times (initially,
and periodically until the old host is taken offline). Both processes
are done initially, then you have the 2nd process migrate any straggling
mail that comes in.

I don't recall you mentioning whether your clients use imap or pop or
both. The logistics can be a little different depending on which they use.

HTH.

--
-Eric 'shubes'

On 10/04/2011 01:18 AM, Casey wrote:
> Ok guys, here is my next challenge...really, the biggest problem I'm
> foreseeing at the moment:
>
> So here I am joyfully (as I curse at the computer under my breath every
> time I find a new problem...) migrating all 4000+ users of the 392
> different domains using the migrate-domain script (if you aren't
> familiar, it basically creates a tarball of every mailbox belonging to
> the domain, and then generates a script that when run will recreate the
> domain, user accounts, mailing lists and forwards --with a minor amount
> of fuss after the fact on the new system) --- so that is all good and
> well, but here is where the problem comes in...I have 165GB of data to
> migrate, and in the end I plan on taking down "Server A" and having
> "Server B" completely take over, except every second after data has been
> migrated over to "Server B", there is a pretty good chance that new mail
> is coming in for one of the 392 different domains.
>
> The problem is that as long as that old server is still up, I need to
> worry about keeping the mail content consistent between the old server
> and the new. Obviously I'd like to have the time between domain
> migration and server deployment down to a minimum, but some of my
> customers are quite large (600 or so email accounts, each of which will
> need to be reconfigured on their end at minimum just to make sure they
> are using smtp-auth, and probably changing the incoming/outgoing server
> settings on a few systems that are statically set to use the server's IP
> versus hostname.
>
> Normally I'd probably opt to use rsync to keep the new server in-tune
> with the old one, but I'm not sure that is exactly an option, because
> the old system has vpopmail compiled with "many-domains" enabled, in
> addition to "big user dirs", so once again I'm trying to interface with
> a different folder structure -- for example, on the old system we have
> domains that are broken up in the domain's directory by number (i.e.;
> domains/0/abc.com, domains/1/bca.com, domains/cad.com, etc) as well as
> users broken up by number in a similar fashion (i.e.;
> domains/abc.com/0/postmaster, domains/1/bca.com/demo,
> domains/cad.com/2/test, etc). Since I'm trying to standardize, I don'
> want to have to compile those features in...but that makes it difficult
> to use something such as rsync to run against a domain folder with the
> flags set that will delete files from the destination that don't exist
> on the host. So what do I do? I need a fairly automated way to keep the
> mail synchronized between the servers before we take the old one down.
> At the moment, there is enough manual intervention required in migrating
> the domains, that it doesn't seem realistic to try to get all the data
> migrated and current the day that we decide to do all of this.
>
> I'm really really really hoping that someone out there has some ideas
> and can help me out with this one. I'd like the least amount of service
> disruption possible, and can't run the risk of missing or losing email
> for my customers. I want to get this to work, because my next project is
> to setup a replicated qmailtoaster, which should be much easier to work
> with and can use rsync and/or unison to get things in check.
>
> There has gotta be a few Toaster Master's out there that have the
> perfect solution...all I need is one good idea ;-)
>
> Thanks in advance,
>
>
> Casey
>
> Smile Global Technical Support
> Submit or check trouble tickets http://billing.smileglobal.com
> www.smileglobal.com <http://www.smileglobal.com>
>


---------------------------------------------------------------------------------
Qmailtoaster is sponsored by Vickers Consulting Group (www.vickersconsulting.com)
Vickers Consulting Group offers Qmailtoaster support and installations.
If you need professional help with your setup, contact them today!
c***@smileglobal.com
2011-10-04 15:36:46 UTC
Permalink
Thanks Eric. They use a little bit of everything. Some customers are
strictly webmail, some use pop and some use imap.

Now that those are in the mix, what do you think?

Connected by DROID on Verizon Wireless

-----Original message-----
From: Eric Shubert <***@shubes.net>
To: qmailtoaster-***@qmailtoaster.com
Sent: Tue, Oct 4, 2011 14:59:40 GMT+00:00
Subject: [qmailtoaster] Re: Migrating from Qmail Rocks install on Solaris to
Qmail Toaster on CentOS

As I see it, you could use 2 different processes (or bridges) for
migration. One process migrates the accounts, and is done only one time.
The 2nd process migrates the mail, and is done several times (initially,
and periodically until the old host is taken offline). Both processes
are done initially, then you have the 2nd process migrate any straggling
mail that comes in.

I don't recall you mentioning whether your clients use imap or pop or
both. The logistics can be a little different depending on which they use.

HTH.

--
-Eric 'shubes'

On 10/04/2011 01:18 AM, Casey wrote:
> Ok guys, here is my next challenge...really, the biggest problem I'm
> foreseeing at the moment:
>
> So here I am joyfully (as I curse at the computer under my breath every
> time I find a new problem...) migrating all 4000+ users of the 392
> different domains using the migrate-domain script (if you aren't
> familiar, it basically creates a tarball of every mailbox belonging to
> the domain, and then generates a script that when run will recreate the
> domain, user accounts, mailing lists and forwards --with a minor amount
> of fuss after the fact on the new system) --- so that is all good and
> well, but here is where the problem comes in...I have 165GB of data to
> migrate, and in the end I plan on taking down "Server A" and having
> "Server B" completely take over, except every second after data has been
> migrated over to "Server B", there is a pretty good chance that new mail
> is coming in for one of the 392 different domains.
>
> The problem is that as long as that old server is still up, I need to
> worry about keeping the mail content consistent between the old server
> and the new. Obviously I'd like to have the time between domain
> migration and server deployment down to a minimum, but some of my
> customers are quite large (600 or so email accounts, each of which will
> need to be reconfigured on their end at minimum just to make sure they
> are using smtp-auth, and probably changing the incoming/outgoing server
> settings on a few systems that are statically set to use the server's IP
> versus hostname.
>
> Normally I'd probably opt to use rsync to keep the new server in-tune
> with the old one, but I'm not sure that is exactly an option, because
> the old system has vpopmail compiled with "many-domains" enabled, in
> addition to "big user dirs", so once again I'm trying to interface with
> a different folder structure -- for example, on the old system we have
> domains that are broken up in the domain's directory by number (i.e.;
> domains/0/abc.com, domains/1/bca.com, domains/cad.com, etc) as well as
> users broken up by number in a similar fashion (i.e.;
> domains/abc.com/0/postmaster, domains/1/bca.com/demo,
> domains/cad.com/2/test, etc). Since I'm trying to standardize, I don'
> want to have to compile those features in...but that makes it difficult
> to use something such as rsync to run against a domain folder with the
> flags set that will delete files from the destination that don't exist
> on the host. So what do I do? I need a fairly automated way to keep the
> mail synchronized between the servers before we take the old one down.
> At the moment, there is enough manual intervention required in migrating
> the domains, that it doesn't seem realistic to try to get all the data
> migrated and current the day that we decide to do all of this.
>
> I'm really really really hoping that someone out there has some ideas
> and can help me out with this one. I'd like the least amount of service
> disruption possible, and can't run the risk of missing or losing email
> for my customers. I want to get this to work, because my next project is
> to setup a replicated qmailtoaster, which should be much easier to work
> with and can use rsync and/or unison to get things in check.
>
> There has gotta be a few Toaster Master's out there that have the
> perfect solution...all I need is one good idea ;-)
>
> Thanks in advance,
>
>
> Casey
>
> Smile Global Technical Support
> Submit or check trouble tickets http://billing.smileglobal.com
> www.smileglobal.com <http://www.smileglobal.com>
>


-----------------------------------------------------------------------------
----
Qmailtoaster is sponsored by Vickers Consulting Group
(www.vickersconsulting.com)
Vickers Consulting Group offers Qmailtoaster support and installations.
If you need professional help with your setup, contact them today!
-----------------------------------------------------------------------------
----
Please visit qmailtoaster.com for the latest news, updates, and
packages.

To unsubscribe, e-mail: qmailtoaster-list-***@qmailtoaster.com
For additional commands, e-mail:
qmailtoaster-list-***@qmailtoaster.com
Eric Shubert
2011-10-04 16:01:02 UTC
Permalink
webmail is imap, so that simplifies a little.

The big/first thing is to freeze any account changes before beginning
the migration. That's pretty much a must, and shouldn't be a big deal.

On the inbound side of things, are you planning to put everything in the
QMT host and eliminate your present gateway and SA hosts? That will
dictate how you'll go about migrating (dns vs routing change).

On the MUA/retrieval side of things, you'll need to have both the old
and the new operational for a period of time. Even after you change DNS
to point to the new host, it takes a while for the change to propagate,
so you'll have requests coming to both servers. After a day or so, you
can turn off imap/pop services on the old host and do a final data
migration to pick up any straggling emails.

BTW, I hope you're using dovecot on your new QMT host. If you haven't
done this yet, take the time to do it now. It's much more efficient than
courier, and does a great job of cleaning things up automatically that
may not be quite right, like index files and such. I would use dovecot
for both imap and pop. Check the wiki regarding dovecot.

--
-Eric 'shubes'


On 10/04/2011 08:36 AM, ***@smileglobal.com wrote:
> Thanks Eric. They use a little bit of everything. Some customers are
> strictly webmail, some use pop and some use imap.
>
> Now that those are in the mix, what do you think?
>
> /Connected by DROID on Verizon Wireless/
>
>
> -----Original message-----
>
> *From: *Eric Shubert <***@shubes.net>*
> To: *qmailtoaster-***@qmailtoaster.com*
> Sent: *Tue, Oct 4, 2011 14:59:40 GMT+00:00*
> Subject: *[qmailtoaster] Re: Migrating from Qmail Rocks install on
> Solaris to Qmail Toaster on CentOS
>
> As I see it, you could use 2 different processes (or bridges) for
> migration. One process migrates the accounts, and is done only one
> time.
> The 2nd process migrates the mail, and is done several times
> (initially,
> and periodically until the old host is taken offline). Both processes
> are done initially, then you have the 2nd process migrate any
> straggling
> mail that comes in.
>
> I don't recall you mentioning whether your clients use imap or pop or
> both. The logistics can be a little different depending on which
> they use.
>
> HTH.
>
> --
> -Eric 'shubes'
>
> On 10/04/2011 01:18 AM, Casey wrote:
> > Ok guys, here is my next challenge...really, the biggest problem I'm
> > foreseeing at the moment:
> >
> > So here I am joyfully (as I curse at the computer under my breath
> every
> > time I find a new problem...) migrating all 4000+ users of the 392
> > different domains using the migrate-domain script (if you aren't
> > familiar, it basically creates a tarball of every mailbox
> belonging to
> > the domain, and then generates a script that when run will
> recreate the
> > domain, user accounts, mailing lists and forwards --with a minor
> amount
> > of fuss after the fact on the new system) --- so that is all good and
> > well, but here is where the problem comes in...I have 165GB of
> data to
> > migrate, and in the end I plan on taking down "Server A" and having
> > "Server B" completely take over, except every second after data
> has been
> > migrated over to "Server B", there is a pretty good chance that
> new mail
> > is coming in for one of the 392 different domains.
> >
> > The problem is that as long as that old server is still up, I need to
> > worry about keeping the mail content consistent between the old
> server
> > and the new. Obviously I'd like to have the time between domain
> > migration and server deployment down to a minimum, but some of my
> > customers are quite large (600 or so email accounts, each of
> which will
> > need to be reconfigured on their end at minimum just to make sure
> they
> > are using smtp-auth, and probably changing the incoming/outgoing
> server
> > settings on a few systems that are statically set to use the
> server's IP
> > versus hostname.
> >
> > Normally I'd probably opt to use rsync to keep the new server in-tune
> > with the old one, but I'm not sure that is exactly an option, because
> > the old system has vpopmail compiled with "many-domains" enabled, in
> > addition to "big user dirs", so once again I'm trying to
> interface with
> > a different folder structure -- for example, on the old system we
> have
> > domains that are broken up in the domain's directory by number (i.e.;
> > domains/0/abc.com, domains/1/bca.com, domains/cad.com, etc) as
> well as
> > users broken up by number in a similar fashion (i.e.;
> > domains/abc.com/0/postmaster, domains/1/bca.com/demo,
> > domains/cad.com/2/test, etc). Since I'm trying to standardize, I don'
> > want to have to compile those features in...but that makes it
> difficult
> > to use something such as rsync to run against a domain folder
> with the
> > flags set that will delete files from the destination that don't
> exist
> > on the host. So what do I do? I need a fairly automated way to
> keep the
> > mail synchronized between the servers before we take the old one
> down.
> > At the moment, there is enough manual intervention required in
> migrating
> > the domains, that it doesn't seem realistic to try to get all the
> data
> > migrated and current the day that we decide to do all of this.
> >
> > I'm really really really hoping that someone out there has some ideas
> > and can help me out with this one. I'd like the least amount of
> service
> > disruption possible, and can't run the risk of missing or losing
> email
> > for my customers. I want to get this to work, because my next
> project is
> > to setup a replicated qmailtoaster, which should be much easier
> to work
> > with and can use rsync and/or unison to get things in check.
> >
> > There has gotta be a few Toaster Master's out there that have the
> > perfect solution...all I need is one good idea ;-)
> >
> > Thanks in advance,
> >
> >
> > Casey
> >
> > Smile Global Technical Support
> > Submit or check trouble tickets http://billing.smileglobal.com
> > www.smileglobal.com <http://www.smileglobal.com>
> >
>
>
> ---------------------------------------------------------------------------------
> Qmailtoaster is sponsored by Vickers Consulting Group
> (www.vickersconsulting.com)
> Vickers Consulting Group offers Qmailtoaster support and installations.
> If you need professional help with your setup, contact them today!
> ---------------------------------------------------------------------------------
> Please visit qmailtoaster.com for the latest news, updates, and
> packages.
>
> To unsubscribe, e-mail: qmailtoaster-list-***@qmailtoaster.com
> For additional commands, e-mail: qmailtoaster-list-***@qmailtoaster.com
>
>



---------------------------------------------------------------------------------
Qmailtoaster is sponsored by Vickers Consulting Group (www.vickersconsulting.com)
Vickers Consulting Group offers Qmailtoaster support and installations.
If you need professional help with your setup, contact them today!
c***@smileglobal.com
2011-10-04 16:42:22 UTC
Permalink
For now no, the gateway and sa hosts will stay put.

Understandable about keeping the old server up while changes propagate.

The old server is using courier, but I guess dovecot could be a better
route. I've used it a little in the past on a different server. What kind of
difficulty am I looking at to switch from courier to dovecot? Also, how
would you recommend that I keep the maildirs up to date? Im trying to figure
out the best way to do that without causing either lost or duplicate mail.
Am I going to want to create tarballs of mailboxes and run some type of
conversion on them before restoring them, or utilize sync, or is there a
better way?

Thanks for all the help you've been offering...I certainly appreciate it.

Connected by DROID on Verizon Wireless

-----Original message-----
From: Eric Shubert <***@shubes.net>
To: qmailtoaster-***@qmailtoaster.com
Sent: Tue, Oct 4, 2011 16:09:28 GMT+00:00
Subject: [qmailtoaster] Re: Migrating from Qmail Rocks install on Solaris to
Qmail Toaster on CentOS

webmail is imap, so that simplifies a little.

The big/first thing is to freeze any account changes before beginning
the migration. That's pretty much a must, and shouldn't be a big deal.

On the inbound side of things, are you planning to put everything in the
QMT host and eliminate your present gateway and SA hosts? That will
dictate how you'll go about migrating (dns vs routing change).

On the MUA/retrieval side of things, you'll need to have both the old
and the new operational for a period of time. Even after you change DNS
to point to the new host, it takes a while for the change to propagate,
so you'll have requests coming to both servers. After a day or so, you
can turn off imap/pop services on the old host and do a final data
migration to pick up any straggling emails.

BTW, I hope you're using dovecot on your new QMT host. If you haven't
done this yet, take the time to do it now. It's much more efficient than
courier, and does a great job of cleaning things up automatically that
may not be quite right, like index files and such. I would use dovecot
for both imap and pop. Check the wiki regarding dovecot.

--
-Eric 'shubes'


On 10/04/2011 08:36 AM, ***@smileglobal.com wrote:
> Thanks Eric. They use a little bit of everything. Some customers are
> strictly webmail, some use pop and some use imap.
>
> Now that those are in the mix, what do you think?
>
> /Connected by DROID on Verizon Wireless/
>
>
> -----Original message-----
>
> *From: *Eric Shubert <***@shubes.net>*
> To: *qmailtoaster-***@qmailtoaster.com*
> Sent: *Tue, Oct 4, 2011 14:59:40 GMT+00:00*
> Subject: *[qmailtoaster] Re: Migrating from Qmail Rocks install on
> Solaris to Qmail Toaster on CentOS
>
> As I see it, you could use 2 different processes (or bridges) for
> migration. One process migrates the accounts, and is done only one
> time.
> The 2nd process migrates the mail, and is done several times
> (initially,
> and periodically until the old host is taken offline). Both processes
> are done initially, then you have the 2nd process migrate any
> straggling
> mail that comes in.
>
> I don't recall you mentioning whether your clients use imap or pop or
> both. The logistics can be a little different depending on which
> they use.
>
> HTH.
>
> --
> -Eric 'shubes'
>
> On 10/04/2011 01:18 AM, Casey wrote:
> > Ok guys, here is my next challenge...really, the biggest problem
I'm
> > foreseeing at the moment:
> >
> > So here I am joyfully (as I curse at the computer under my breath
> every
> > time I find a new problem...) migrating all 4000+ users of the 392
> > different domains using the migrate-domain script (if you aren't
> > familiar, it basically creates a tarball of every mailbox
> belonging to
> > the domain, and then generates a script that when run will
> recreate the
> > domain, user accounts, mailing lists and forwards --with a minor
> amount
> > of fuss after the fact on the new system) --- so that is all good
and
> > well, but here is where the problem comes in...I have 165GB of
> data to
> > migrate, and in the end I plan on taking down "Server A" and having
> > "Server B" completely take over, except every second after data
> has been
> > migrated over to "Server B", there is a pretty good chance that
> new mail
> > is coming in for one of the 392 different domains.
> >
> > The problem is that as long as that old server is still up, I need
to
> > worry about keeping the mail content consistent between the old
> server
> > and the new. Obviously I'd like to have the time between domain
> > migration and server deployment down to a minimum, but some of my
> > customers are quite large (600 or so email accounts, each of
> which will
> > need to be reconfigured on their end at minimum just to make sure
> they
> > are using smtp-auth, and probably changing the incoming/outgoing
> server
> > settings on a few systems that are statically set to use the
> server's IP
> > versus hostname.
> >
> > Normally I'd probably opt to use rsync to keep the new server
in-tune
> > with the old one, but I'm not sure that is exactly an option,
because
> > the old system has vpopmail compiled with "many-domains" enabled,
in
> > addition to "big user dirs", so once again I'm trying to
> interface with
> > a different folder structure -- for example, on the old system we
> have
> > domains that are broken up in the domain's directory by number
(i.e.;
> > domains/0/abc.com, domains/1/bca.com, domains/cad.com, etc) as
> well as
> > users broken up by number in a similar fashion (i.e.;
> > domains/abc.com/0/postmaster, domains/1/bca.com/demo,
> > domains/cad.com/2/test, etc). Since I'm trying to standardize, I
don'
> > want to have to compile those features in...but that makes it
> difficult
> > to use something such as rsync to run against a domain folder
> with the
> > flags set that will delete files from the destination that don't
> exist
> > on the host. So what do I do? I need a fairly automated way to
> keep the
> > mail synchronized between the servers before we take the old one
> down.
> > At the moment, there is enough manual intervention required in
> migrating
> > the domains, that it doesn't seem realistic to try to get all the
> data
> > migrated and current the day that we decide to do all of this.
> >
> > I'm really really really hoping that someone out there has some
ideas
> > and can help me out with this one. I'd like the least amount of
> service
> > disruption possible, and can't run the risk of missing or losing
> email
> > for my customers. I want to get this to work, because my next
> project is
> > to setup a replicated qmailtoaster, which should be much easier
> to work
> > with and can use rsync and/or unison to get things in check.
> >
> > There has gotta be a few Toaster Master's out there that have the
> > perfect solution...all I need is one good idea ;-)
> >
> > Thanks in advance,
> >
> >
> > Casey
> >
> > Smile Global Technical Support
> > Submit or check trouble tickets http://billing.smileglobal.com
> > www.smileglobal.com <http://www.smileglobal.com>
> >
>
>
>
-----------------------------------------------------------------------------
----
> Qmailtoaster is sponsored by Vickers Consulting Group
> (www.vickersconsulting.com)
> Vickers Consulting Group offers Qmailtoaster support and
installations.
> If you need professional help with your setup, contact them today!
>
-----------------------------------------------------------------------------
----
> Please visit qmailtoaster.com for the latest news, updates, and
> packages.
>
> To unsubscribe, e-mail: qmailtoaster-list-***@qmailtoaster.com
> For additional commands, e-mail:
qmailtoaster-list-***@qmailtoaster.com
>
>



-----------------------------------------------------------------------------
----
Qmailtoaster is sponsored by Vickers Consulting Group
(www.vickersconsulting.com)
Vickers Consulting Group offers Qmailtoaster support and installations.
If you need professional help with your setup, contact them today!
-----------------------------------------------------------------------------
----
Please visit qmailtoaster.com for the latest news, updates, and
packages.

To unsubscribe, e-mail: qmailtoaster-list-***@qmailtoaster.com
For additional commands, e-mail:
qmailtoaster-list-***@qmailtoaster.com
Eric Shubert
2011-10-04 17:29:36 UTC
Permalink
On 10/04/2011 09:42 AM, ***@smileglobal.com wrote:
> For now no, the gateway and sa hosts will stay put.

Then you can simply change the smtproutes in the sa host to point to the
new server at an appropriate time, after the new host is up and running
and accounts exist there.

> Understandable about keeping the old server up while changes propagate.

Don't need to worry about the inbound side though, as that'll stay the same.

> The old server is using courier, but I guess dovecot could be a better
> route. I've used it a little in the past on a different server. What
> kind of difficulty am I looking at to switch from courier to dovecot?

It's not tough at all. See the wiki for instructions. There's an rpm for
QMT available in the QTP repo.

> Also, how would you recommend that I keep the maildirs up to date? Im
> trying to figure out the best way to do that without causing either lost
> or duplicate mail. Am I going to want to create tarballs of mailboxes
> and run some type of conversion on them before restoring them, or
> utilize sync, or is there a better way?

I think you're gonna need to do some scripting for this, as the
directory structure is changing. Basically, you want something that'll
migrate the maildir folders from one host to another. I think I'd use
tar with a couple scripts to 'drive' the thing.

On the old side, the script should find a domain (or drive it with a
list), then find each account, then tarball up the maildir into a single
file. At the end of the process, you can tarball up all the account
tarballs into a single archive for transfer.

After transfer, on the new side, untar the transferred file, and another
script would process each maildir tarball, and untar it into the
appropriate place. Be careful not to wipe out any mail that may have
come in directly to the new server already.

The only problem I can think of is if you have pop users who leave
messages on the server. Those are going to be duplicated. If your users
can't live with that, you'll need to do some extra work to get that part
taken care of.

> Thanks for all the help you've been offering...I certainly appreciate it.

Sure.
I hope you're taking copious notes for the wiki. ;)

--
-Eric 'shubes'


---------------------------------------------------------------------------------
Qmailtoaster is sponsored by Vickers Consulting Group (www.vickersconsulting.com)
Vickers Consulting Group offers Qmailtoaster support and installations.
If you need professional help with your setup, contact them today!
Joel Eddy
2011-10-04 17:41:42 UTC
Permalink
This is only my 2 cents.
You will want to lower your DNS TTL down to 3600 before you start this
process at least 3 or 4 days before.

That way when you transfer over you won't get as many phones call
complaining about mail issues. And it should clear itself in about an hour.

TTFN

Joel

-----Original Message-----
From: Eric Shubert [mailto:***@shubes.net]
Sent: Tuesday, October 04, 2011 12:30 PM
To: qmailtoaster-***@qmailtoaster.com
Subject: [qmailtoaster] Re: Migrating from Qmail Rocks install on Solaris to
Qmail Toaster on CentOS

On 10/04/2011 09:42 AM, ***@smileglobal.com wrote:
> For now no, the gateway and sa hosts will stay put.

Then you can simply change the smtproutes in the sa host to point to the
new server at an appropriate time, after the new host is up and running
and accounts exist there.

> Understandable about keeping the old server up while changes propagate.

Don't need to worry about the inbound side though, as that'll stay the same.

> The old server is using courier, but I guess dovecot could be a better
> route. I've used it a little in the past on a different server. What
> kind of difficulty am I looking at to switch from courier to dovecot?

It's not tough at all. See the wiki for instructions. There's an rpm for
QMT available in the QTP repo.

> Also, how would you recommend that I keep the maildirs up to date? Im
> trying to figure out the best way to do that without causing either lost
> or duplicate mail. Am I going to want to create tarballs of mailboxes
> and run some type of conversion on them before restoring them, or
> utilize sync, or is there a better way?

I think you're gonna need to do some scripting for this, as the
directory structure is changing. Basically, you want something that'll
migrate the maildir folders from one host to another. I think I'd use
tar with a couple scripts to 'drive' the thing.

On the old side, the script should find a domain (or drive it with a
list), then find each account, then tarball up the maildir into a single
file. At the end of the process, you can tarball up all the account
tarballs into a single archive for transfer.

After transfer, on the new side, untar the transferred file, and another
script would process each maildir tarball, and untar it into the
appropriate place. Be careful not to wipe out any mail that may have
come in directly to the new server already.

The only problem I can think of is if you have pop users who leave
messages on the server. Those are going to be duplicated. If your users
can't live with that, you'll need to do some extra work to get that part
taken care of.

> Thanks for all the help you've been offering...I certainly appreciate it.

Sure.
I hope you're taking copious notes for the wiki. ;)

--
-Eric 'shubes'


----------------------------------------------------------------------------
-----
Qmailtoaster is sponsored by Vickers Consulting Group
(www.vickersconsulting.com)
Vickers Consulting Group offers Qmailtoaster support and installations.
If you need professional help with your setup, contact them today!
----------------------------------------------------------------------------
-----
Please visit qmailtoaster.com for the latest news, updates, and
packages.

To unsubscribe, e-mail: qmailtoaster-list-***@qmailtoaster.com
For additional commands, e-mail:
qmailtoaster-list-***@qmailtoaster.com



-----
No virus found in this message.
Checked by AVG - www.avg.com
Version: 2012.0.1809 / Virus Database: 2085/4537 - Release Date: 10/04/11


---------------------------------------------------------------------------------
Qmailtoaster is sponsored by Vickers Consulting Group (www.vickersconsulting.com)
Vickers Consulting Group offers Qmailtoaster support and installations.
If you need professional help with your setup, contact them today!
Eric Shubert
2011-10-04 18:00:07 UTC
Permalink
Good advice, Joel. This is a good thing to do whenever you're changing
DNS around. I might even go with 600. ;)

Casey has a lot of domains, but it's probably still worth doing. Users
don't understand "it's already fixed, but it will take some time for
your system to work". ;)

--
-Eric 'shubes'

On 10/04/2011 10:41 AM, Joel Eddy wrote:
> This is only my 2 cents.
> You will want to lower your DNS TTL down to 3600 before you start this
> process at least 3 or 4 days before.
>
> That way when you transfer over you won't get as many phones call
> complaining about mail issues. And it should clear itself in about an hour.
>
> TTFN
>
> Joel
>


---------------------------------------------------------------------------------
Qmailtoaster is sponsored by Vickers Consulting Group (www.vickersconsulting.com)
Vickers Consulting Group offers Qmailtoaster support and installations.
If you need professional help with your setup, contact them today!
Joel Eddy
2011-10-04 18:12:14 UTC
Permalink
Most definitively. It will save a ton of phone calls.

Joel

-----Original Message-----
From: Eric Shubert [mailto:***@shubes.net]
Sent: Tuesday, October 04, 2011 1:00 PM
To: qmailtoaster-***@qmailtoaster.com
Subject: [qmailtoaster] Re: Migrating from Qmail Rocks install on Solaris to
Qmail Toaster on CentOS

Good advice, Joel. This is a good thing to do whenever you're changing
DNS around. I might even go with 600. ;)

Casey has a lot of domains, but it's probably still worth doing. Users
don't understand "it's already fixed, but it will take some time for
your system to work". ;)

--
-Eric 'shubes'

On 10/04/2011 10:41 AM, Joel Eddy wrote:
> This is only my 2 cents.
> You will want to lower your DNS TTL down to 3600 before you start this
> process at least 3 or 4 days before.
>
> That way when you transfer over you won't get as many phones call
> complaining about mail issues. And it should clear itself in about an
hour.
>
> TTFN
>
> Joel
>


----------------------------------------------------------------------------
-----
Qmailtoaster is sponsored by Vickers Consulting Group
(www.vickersconsulting.com)
Vickers Consulting Group offers Qmailtoaster support and installations.
If you need professional help with your setup, contact them today!
----------------------------------------------------------------------------
-----
Please visit qmailtoaster.com for the latest news, updates, and
packages.

To unsubscribe, e-mail: qmailtoaster-list-***@qmailtoaster.com
For additional commands, e-mail:
qmailtoaster-list-***@qmailtoaster.com



-----
No virus found in this message.
Checked by AVG - www.avg.com
Version: 2012.0.1809 / Virus Database: 2085/4537 - Release Date: 10/04/11


---------------------------------------------------------------------------------
Qmailtoaster is sponsored by Vickers Consulting Group (www.vickersconsulting.com)
Vickers Consulting Group offers Qmailtoaster support and installations.
If you need professional help with your setup, contact them today!
Casey
2011-10-04 19:04:52 UTC
Permalink
Thanks Joel. That is a great point. Definitely add that to my checklist ;-)

Casey

Smile Global Technical Support
Submit or check trouble tickets http://billing.smileglobal.com
www.smileglobal.com <http://www.smileglobal.com>

On 10/4/11 10:41 AM, Joel Eddy wrote:
> This is only my 2 cents.
> You will want to lower your DNS TTL down to 3600 before you start this
> process at least 3 or 4 days before.
>
> That way when you transfer over you won't get as many phones call
> complaining about mail issues. And it should clear itself in about an hour.
>
> TTFN
>
> Joel
>
> -----Original Message-----
> From: Eric Shubert [mailto:***@shubes.net]
> Sent: Tuesday, October 04, 2011 12:30 PM
> To: qmailtoaster-***@qmailtoaster.com
> Subject: [qmailtoaster] Re: Migrating from Qmail Rocks install on Solaris to
> Qmail Toaster on CentOS
>
> On 10/04/2011 09:42 AM, ***@smileglobal.com wrote:
>> For now no, the gateway and sa hosts will stay put.
> Then you can simply change the smtproutes in the sa host to point to the
> new server at an appropriate time, after the new host is up and running
> and accounts exist there.
>
>> Understandable about keeping the old server up while changes propagate.
> Don't need to worry about the inbound side though, as that'll stay the same.
>
>> The old server is using courier, but I guess dovecot could be a better
>> route. I've used it a little in the past on a different server. What
>> kind of difficulty am I looking at to switch from courier to dovecot?
> It's not tough at all. See the wiki for instructions. There's an rpm for
> QMT available in the QTP repo.
>
>> Also, how would you recommend that I keep the maildirs up to date? Im
>> trying to figure out the best way to do that without causing either lost
>> or duplicate mail. Am I going to want to create tarballs of mailboxes
>> and run some type of conversion on them before restoring them, or
>> utilize sync, or is there a better way?
> I think you're gonna need to do some scripting for this, as the
> directory structure is changing. Basically, you want something that'll
> migrate the maildir folders from one host to another. I think I'd use
> tar with a couple scripts to 'drive' the thing.
>
> On the old side, the script should find a domain (or drive it with a
> list), then find each account, then tarball up the maildir into a single
> file. At the end of the process, you can tarball up all the account
> tarballs into a single archive for transfer.
>
> After transfer, on the new side, untar the transferred file, and another
> script would process each maildir tarball, and untar it into the
> appropriate place. Be careful not to wipe out any mail that may have
> come in directly to the new server already.
>
> The only problem I can think of is if you have pop users who leave
> messages on the server. Those are going to be duplicated. If your users
> can't live with that, you'll need to do some extra work to get that part
> taken care of.
>
>> Thanks for all the help you've been offering...I certainly appreciate it.
> Sure.
> I hope you're taking copious notes for the wiki. ;)
>
Casey
2011-10-05 02:51:00 UTC
Permalink
Eric,

I looked over the Dovecot article on the wiki...seems straightforward
enough, but the one thing I was wondering - should I proceed with
migrating everything as I am now and then when I have the first batch
everything moved over, then install Dovecot and run that migration
script? I'm guessing that I'll need to run that script any time that
maildirs are copied from the old server, correct?

Casey

Smile Global Technical Support
Submit or check trouble tickets http://billing.smileglobal.com
www.smileglobal.com <http://www.smileglobal.com>

On 10/4/11 12:04 PM, Casey wrote:
> Thanks Joel. That is a great point. Definitely add that to my
> checklist ;-)
>
> Casey
>
> Smile Global Technical Support
> Submit or check trouble tickets http://billing.smileglobal.com
> www.smileglobal.com <http://www.smileglobal.com>
>
> On 10/4/11 10:41 AM, Joel Eddy wrote:
>> This is only my 2 cents.
>> You will want to lower your DNS TTL down to 3600 before you start this
>> process at least 3 or 4 days before.
>>
>> That way when you transfer over you won't get as many phones call
>> complaining about mail issues. And it should clear itself in about an hour.
>>
>> TTFN
>>
>> Joel
>>
>> -----Original Message-----
>> From: Eric Shubert [mailto:***@shubes.net]
>> Sent: Tuesday, October 04, 2011 12:30 PM
>> To:qmailtoaster-***@qmailtoaster.com
>> Subject: [qmailtoaster] Re: Migrating from Qmail Rocks install on Solaris to
>> Qmail Toaster on CentOS
>>
>> On 10/04/2011 09:42 AM,***@smileglobal.com wrote:
>>> For now no, the gateway and sa hosts will stay put.
>> Then you can simply change the smtproutes in the sa host to point to the
>> new server at an appropriate time, after the new host is up and running
>> and accounts exist there.
>>
>>> Understandable about keeping the old server up while changes propagate.
>> Don't need to worry about the inbound side though, as that'll stay the same.
>>
>>> The old server is using courier, but I guess dovecot could be a better
>>> route. I've used it a little in the past on a different server. What
>>> kind of difficulty am I looking at to switch from courier to dovecot?
>> It's not tough at all. See the wiki for instructions. There's an rpm for
>> QMT available in the QTP repo.
>>
>>> Also, how would you recommend that I keep the maildirs up to date? Im
>>> trying to figure out the best way to do that without causing either lost
>>> or duplicate mail. Am I going to want to create tarballs of mailboxes
>>> and run some type of conversion on them before restoring them, or
>>> utilize sync, or is there a better way?
>> I think you're gonna need to do some scripting for this, as the
>> directory structure is changing. Basically, you want something that'll
>> migrate the maildir folders from one host to another. I think I'd use
>> tar with a couple scripts to 'drive' the thing.
>>
>> On the old side, the script should find a domain (or drive it with a
>> list), then find each account, then tarball up the maildir into a single
>> file. At the end of the process, you can tarball up all the account
>> tarballs into a single archive for transfer.
>>
>> After transfer, on the new side, untar the transferred file, and another
>> script would process each maildir tarball, and untar it into the
>> appropriate place. Be careful not to wipe out any mail that may have
>> come in directly to the new server already.
>>
>> The only problem I can think of is if you have pop users who leave
>> messages on the server. Those are going to be duplicated. If your users
>> can't live with that, you'll need to do some extra work to get that part
>> taken care of.
>>
>>> Thanks for all the help you've been offering...I certainly appreciate it.
>> Sure.
>> I hope you're taking copious notes for the wiki. ;)
>>
Eric Shubert
2011-10-05 03:31:54 UTC
Permalink
On 10/04/2011 07:51 PM, Casey wrote:
> Eric,
>
> I looked over the Dovecot article on the wiki...seems straightforward
> enough, but the one thing I was wondering - should I proceed with
> migrating everything as I am now and then when I have the first batch
> everything moved over, then install Dovecot and run that migration
> script?

No, I would install and use dovecot from the get-go, before doing any
migration. The filesystem structure for maildirs is the same.

> I'm guessing that I'll need to run that script any time that
> maildirs are copied from the old server, correct?

Which script would that be?

You can drop emails from wherever (your script) into the Maildir/new
directory anytime/anyway, and dovecot should figure out how to handle
it. I think. Imap-wise anyhow. I don't think pop would be a problem
either, although I haven't exactly done that, to be honest.

--
-Eric 'shubes'


---------------------------------------------------------------------------------
Qmailtoaster is sponsored by Vickers Consulting Group (www.vickersconsulting.com)
Vickers Consulting Group offers Qmailtoaster support and installations.
If you need professional help with your setup, contact them today!
Casey
2011-10-05 03:57:42 UTC
Permalink
I was checking out the following page on the wiki:
http://wiki.qmailtoaster.com/index.php/Replacing_Courier_IMAP_with_Dovecot_IMAP

The 4th section down ("Convert the mailboxes"):

This is the part you really should test. The success or failure of this
part will determine how transparent your migration is to your users.
You'll not likely loose any mail (you have backups, right??), but your
users might end up having to re-subscribe to folders and/or re-download
email headers, depending on the client software they use.

First we'll get the conversion script:

# cd /usr/local/sbin
# wgethttp://www.dovecot.org/tools/courier-dovecot-migrate.pl
...
# chmod 0700 courier-dovecot-migrate.pl
# cd ~vpopmail/domains

This command will do a "dry run" to make sure things will work correctly...

# /usr/local/sbin/courier-dovecot-migrate.pl --recursive

You should see your user accounts scroll by and say something like
"UIDLIST not found" but nothing else. At the end of the script you
should have 0 ERRORS.

This command actually does the conversion.

# /usr/local/sbin/courier-dovecot-migrate.pl --recursive --convert

If everything went well, you can proceed. If not, you might want to post
a message to the qmail-toaster list to get some help.



This is the script I was referring to. Is it necessary to do this?

Sometime in the not-so-distant future I'd like to set this new server up
to replicate to another offsite QMT box...I've purchased and viewed
Jake's Replicated QMT videos and was planning on doing something
similar, however last time I recompiled vpopmail I noticed the
"--enable-mysql-replication" option and wasn't sure if that would be
something I would need to compile in to do the replication or not. The
video doesn't mention it, and the documentation briefly rattles off
something about read-only replication from a master server. Any chance
anyone could provide me with a bit more in-depth explanation?

Last but not least (at least for today...), what about QControl? Is this
something I should look at purchasing? The screenshots make it look
kinda nifty, but I was hoping to someone else' input.

Thanks!

Casey

Smile Global Technical Support
Submit or check trouble tickets http://billing.smileglobal.com
www.smileglobal.com <http://www.smileglobal.com>

On 10/4/11 8:31 PM, Eric Shubert wrote:
> On 10/04/2011 07:51 PM, Casey wrote:
>> Eric,
>>
>> I looked over the Dovecot article on the wiki...seems straightforward
>> enough, but the one thing I was wondering - should I proceed with
>> migrating everything as I am now and then when I have the first batch
>> everything moved over, then install Dovecot and run that migration
>> script?
>
> No, I would install and use dovecot from the get-go, before doing any
> migration. The filesystem structure for maildirs is the same.
>
>> I'm guessing that I'll need to run that script any time that
>> maildirs are copied from the old server, correct?
>
> Which script would that be?
>
> You can drop emails from wherever (your script) into the Maildir/new
> directory anytime/anyway, and dovecot should figure out how to handle
> it. I think. Imap-wise anyhow. I don't think pop would be a problem
> either, although I haven't exactly done that, to be honest.
>
Eric Shubert
2011-10-05 15:39:34 UTC
Permalink
On 10/04/2011 08:57 PM, Casey wrote:
>
> # cd /usr/local/sbin
> # wgethttp://www.dovecot.org/tools/courier-dovecot-migrate.pl
> ...
> # chmod 0700 courier-dovecot-migrate.pl
> # cd ~vpopmail/domains
>
>
> This is the script I was referring to. Is it necessary to do this?

I would run it after the first migration, but not subsequent ones. On
cleanup runs, simply put mail files on the new host and let it go at that.

--
-Eric 'shubes'


---------------------------------------------------------------------------------
Qmailtoaster is sponsored by Vickers Consulting Group (www.vickersconsulting.com)
Vickers Consulting Group offers Qmailtoaster support and installations.
If you need professional help with your setup, contact them today!
Casey
2011-10-06 22:08:18 UTC
Permalink
Will do. On another track...what do you have to say about QControl? For
someone with so many users, would it be worth switching over to QControl
from Vqadmin?

Thanks,

Casey

Smile Global Technical Support
Submit or check trouble tickets http://billing.smileglobal.com
www.smileglobal.com <http://www.smileglobal.com>

On 10/5/11 8:39 AM, Eric Shubert wrote:
> On 10/04/2011 08:57 PM, Casey wrote:
>>
>> # cd /usr/local/sbin
>> # wgethttp://www.dovecot.org/tools/courier-dovecot-migrate.pl
>> ...
>> # chmod 0700 courier-dovecot-migrate.pl
>> # cd ~vpopmail/domains
>>
>>
>> This is the script I was referring to. Is it necessary to do this?
>
> I would run it after the first migration, but not subsequent ones. On
> cleanup runs, simply put mail files on the new host and let it go at
> that.
>
Eric Shubert
2011-10-06 23:49:38 UTC
Permalink
If you need what vqadmin provides in a gui and you need it to work right
away, I'd go ahead and get QControl. Personally, I don't use vqadmin at
all, because it's broken. I can get by with the CLI for maintaining
domains, and qmailadmin works fine for maintaining users.
(Of course, quotas are still broken, so I just use NOQUOTA for all
accounts). Keep in mind though, I'm not an ISP. I help people host their
own.
--
-Eric 'shubes'

On 10/06/2011 03:08 PM, Casey wrote:
> Will do. On another track...what do you have to say about QControl? For
> someone with so many users, would it be worth switching over to QControl
> from Vqadmin?
>
> Thanks,
>
> Casey
>
> Smile Global Technical Support
> Submit or check trouble tickets http://billing.smileglobal.com
> www.smileglobal.com <http://www.smileglobal.com>
>
> On 10/5/11 8:39 AM, Eric Shubert wrote:
>> On 10/04/2011 08:57 PM, Casey wrote:
>>>
>>> # cd /usr/local/sbin
>>> # wgethttp://www.dovecot.org/tools/courier-dovecot-migrate.pl
>>> ...
>>> # chmod 0700 courier-dovecot-migrate.pl
>>> # cd ~vpopmail/domains
>>>
>>>
>>> This is the script I was referring to. Is it necessary to do this?
>>
>> I would run it after the first migration, but not subsequent ones. On
>> cleanup runs, simply put mail files on the new host and let it go at
>> that.
>>



---------------------------------------------------------------------------------
Qmailtoaster is sponsored by Vickers Consulting Group (www.vickersconsulting.com)
Vickers Consulting Group offers Qmailtoaster support and installations.
If you need professional help with your setup, contact them today!
Casey
2011-10-07 00:32:31 UTC
Permalink
Where exactly are quotas broken? Vqadmin has been handy on the old
server when I'm in a pinch and need to make a quick chance but don't
necessarily have access via SSH. QControl seems like it provides all the
same features plus the stats, which could be somewhat nice.

Another thing I'm curious about, is how to give some of our customers
access to some of the stats that are displayed through isoqlog. One of
our biggest clients has over 600 accounts and their IT department is
interested in seeing some statistics for their domain. Boy it sure would
be nice if there was some kind of open source piece of software that
combined isoqlog as well as the output from qmailmrtg, so the customers
could get a breakdown of mail usage per domain.

Also, going back a couple of emails, I'm still a bit confused about the
"--enable-mysql-replication" flag available when recompiling vpopmail.
Can anyone provide a bit more of an explanation of what that does, and
what kind of a situation in which you would want to enable it?

Oh and I changed my workflow a bit on the migration...I was using the
migrate-domains script pretty much as is to backup each mailbox,
aliases, mailing list, etc and finally generate an "import" script. That
method was definitely working, but then I thought to myself - you know
what, this is going to end up causing alot more work for me, because it
just contributes to the problem of having to keep all of the maildirs in
sync. So what I did was modify the script, and commented out the lines
that created the tarballs of each mailbox, as well as the line that
added the extraction portion to the "import" script. So what ends up
happening is that the mailboxes don't get backed up with the version of
the script, but the full import script is still created, and each line
that would extract the individual mailbox archive is commented out, so
the script simply creates the domain & postmaster account, as well as
each individual account, alias, and mailing list, in addition to setting
the parameters such as password, quota, comment, etc. So at least this
will speed up the process of getting everyone added into the system.
Then i can run the original script on the old server to backup the
mailboxes, rsync them over, and use sed to modify each domain's import
script -- uncomment the mailbox extraction line, and batta bing batta
boom...that should do the trick (this all sounded great in my head and
in an email seems alright, so I'm sure in the field there will be some
form of minor hiccup or two)

Thanks!

Casey

Smile Global Technical Support
Submit or check trouble tickets http://billing.smileglobal.com
www.smileglobal.com <http://www.smileglobal.com>

On 10/6/11 4:49 PM, Eric Shubert wrote:
> If you need what vqadmin provides in a gui and you need it to work
> right away, I'd go ahead and get QControl. Personally, I don't use
> vqadmin at all, because it's broken. I can get by with the CLI for
> maintaining domains, and qmailadmin works fine for maintaining users.
> (Of course, quotas are still broken, so I just use NOQUOTA for all
> accounts). Keep in mind though, I'm not an ISP. I help people host
> their own.
Eric Shubert
2011-10-07 01:03:18 UTC
Permalink
On 10/06/2011 05:32 PM, Casey wrote:
> Where exactly are quotas broken?

If I knew, I might fix it. ;)
I don't know, but sometimes it reports people as being over quota when
they're not.

> Vqadmin has been handy on the old
> server when I'm in a pinch and need to make a quick chance but don't
> necessarily have access via SSH. QControl seems like it provides all the
> same features plus the stats, which could be somewhat nice.

QControl *is* nice.

> Another thing I'm curious about, is how to give some of our customers
> access to some of the stats that are displayed through isoqlog. One of
> our biggest clients has over 600 accounts and their IT department is
> interested in seeing some statistics for their domain. Boy it sure would
> be nice if there was some kind of open source piece of software that
> combined isoqlog as well as the output from qmailmrtg, so the customers
> could get a breakdown of mail usage per domain.

Yes, it would. Or, you could give them their own QMT host (virtual
machine). ;)

> Also, going back a couple of emails, I'm still a bit confused about the
> "--enable-mysql-replication" flag available when recompiling vpopmail.
> Can anyone provide a bit more of an explanation of what that does, and
> what kind of a situation in which you would want to enable it?

I don't know about that off hand. Should be documented in vpopmail
somewhere. Anyone else?

> Oh and I changed my workflow a bit on the migration...I was using the
> migrate-domains script pretty much as is to backup each mailbox,
> aliases, mailing list, etc and finally generate an "import" script. That
> method was definitely working, but then I thought to myself - you know
> what, this is going to end up causing alot more work for me, because it
> just contributes to the problem of having to keep all of the maildirs in
> sync. So what I did was modify the script, and commented out the lines
> that created the tarballs of each mailbox, as well as the line that
> added the extraction portion to the "import" script. So what ends up
> happening is that the mailboxes don't get backed up with the version of
> the script, but the full import script is still created, and each line
> that would extract the individual mailbox archive is commented out, so
> the script simply creates the domain & postmaster account, as well as
> each individual account, alias, and mailing list, in addition to setting
> the parameters such as password, quota, comment, etc. So at least this
> will speed up the process of getting everyone added into the system.

That part I follow. Get the accounts migrated at first, and forget the
emails for the time being. Keep in mind, you'll want to freeze any
account activity from that point on, 'til you're operational on the new
host.

> Then i can run the original script on the old server to backup the
> mailboxes, rsync them over, and use sed to modify each domain's import
> script -- uncomment the mailbox extraction line, and batta bing batta
> boom...that should do the trick (this all sounded great in my head and
> in an email seems alright, so I'm sure in the field there will be some
> form of minor hiccup or two)

I'm not sure I follow here, but ok. I don't think you want to process
anything account-wise at this stage (counting on a re-add not to mess
anything up). Plus, I don't see how rsync is buying you anything at this
stage. I think I'd just ssh the tarball(s) over to the new host.

Splitting up the process like this makes sense to me though. Do accounts
once, then maildirs as many times as you need to.

> Thanks!
>
> Casey
>
> Smile Global Technical Support
> Submit or check trouble tickets http://billing.smileglobal.com
> www.smileglobal.com <http://www.smileglobal.com>
>
> On 10/6/11 4:49 PM, Eric Shubert wrote:
>> If you need what vqadmin provides in a gui and you need it to work
>> right away, I'd go ahead and get QControl. Personally, I don't use
>> vqadmin at all, because it's broken. I can get by with the CLI for
>> maintaining domains, and qmailadmin works fine for maintaining users.
>> (Of course, quotas are still broken, so I just use NOQUOTA for all
>> accounts). Keep in mind though, I'm not an ISP. I help people host
>> their own.


--
-Eric 'shubes'


---------------------------------------------------------------------------------
Qmailtoaster is sponsored by Vickers Consulting Group (www.vickersconsulting.com)
Vickers Consulting Group offers Qmailtoaster support and installations.
If you need professional help with your setup, contact them today!
Casey
2011-10-07 02:28:15 UTC
Permalink
Ahh, that explains alot then. I've definitely had the occasional
complaint from a user about being over quota, only to find that they
were far from it.

I'll shell out the cash and take a look at QControl. Can only make
things better, especially considering that the current web-based
alternative is broken ;-)

I suppose for the large customer I could do something like that, but
that just one more server to maintain and worry about fixing when
something goes wrong. I'll do some research and see if there is
something out there I'm missing that could accomplish what it is that
I'm hoping to accomplish.

What is the quickest/easiest/most efficient way of "freezing" account
activity? By this I'm assuming you mean, creating new accounts, changing
passwords, etc?

Casey

Smile Global Technical Support
Submit or check trouble tickets http://billing.smileglobal.com
www.smileglobal.com <http://www.smileglobal.com>

On 10/6/11 6:03 PM, Eric Shubert wrote:
> On 10/06/2011 05:32 PM, Casey wrote:
>> Where exactly are quotas broken?
>
> If I knew, I might fix it. ;)
> I don't know, but sometimes it reports people as being over quota when
> they're not.
>
>> Vqadmin has been handy on the old
>> server when I'm in a pinch and need to make a quick chance but don't
>> necessarily have access via SSH. QControl seems like it provides all the
>> same features plus the stats, which could be somewhat nice.
>
> QControl *is* nice.
>
>> Another thing I'm curious about, is how to give some of our customers
>> access to some of the stats that are displayed through isoqlog. One of
>> our biggest clients has over 600 accounts and their IT department is
>> interested in seeing some statistics for their domain. Boy it sure would
>> be nice if there was some kind of open source piece of software that
>> combined isoqlog as well as the output from qmailmrtg, so the customers
>> could get a breakdown of mail usage per domain.
>
> Yes, it would. Or, you could give them their own QMT host (virtual
> machine). ;)
>
>> Also, going back a couple of emails, I'm still a bit confused about the
>> "--enable-mysql-replication" flag available when recompiling vpopmail.
>> Can anyone provide a bit more of an explanation of what that does, and
>> what kind of a situation in which you would want to enable it?
>
> I don't know about that off hand. Should be documented in vpopmail
> somewhere. Anyone else?
>
>> Oh and I changed my workflow a bit on the migration...I was using the
>> migrate-domains script pretty much as is to backup each mailbox,
>> aliases, mailing list, etc and finally generate an "import" script. That
>> method was definitely working, but then I thought to myself - you know
>> what, this is going to end up causing alot more work for me, because it
>> just contributes to the problem of having to keep all of the maildirs in
>> sync. So what I did was modify the script, and commented out the lines
>> that created the tarballs of each mailbox, as well as the line that
>> added the extraction portion to the "import" script. So what ends up
>> happening is that the mailboxes don't get backed up with the version of
>> the script, but the full import script is still created, and each line
>> that would extract the individual mailbox archive is commented out, so
>> the script simply creates the domain & postmaster account, as well as
>> each individual account, alias, and mailing list, in addition to setting
>> the parameters such as password, quota, comment, etc. So at least this
>> will speed up the process of getting everyone added into the system.
>
> That part I follow. Get the accounts migrated at first, and forget the
> emails for the time being. Keep in mind, you'll want to freeze any
> account activity from that point on, 'til you're operational on the
> new host.
>
>> Then i can run the original script on the old server to backup the
>> mailboxes, rsync them over, and use sed to modify each domain's import
>> script -- uncomment the mailbox extraction line, and batta bing batta
>> boom...that should do the trick (this all sounded great in my head and
>> in an email seems alright, so I'm sure in the field there will be some
>> form of minor hiccup or two)
>
> I'm not sure I follow here, but ok. I don't think you want to process
> anything account-wise at this stage (counting on a re-add not to mess
> anything up). Plus, I don't see how rsync is buying you anything at
> this stage. I think I'd just ssh the tarball(s) over to the new host.
>
> Splitting up the process like this makes sense to me though. Do
> accounts once, then maildirs as many times as you need to.
>
>> Thanks!
>>
>> Casey
>>
>> Smile Global Technical Support
>> Submit or check trouble tickets http://billing.smileglobal.com
>> www.smileglobal.com <http://www.smileglobal.com>
>>
>> On 10/6/11 4:49 PM, Eric Shubert wrote:
>>> If you need what vqadmin provides in a gui and you need it to work
>>> right away, I'd go ahead and get QControl. Personally, I don't use
>>> vqadmin at all, because it's broken. I can get by with the CLI for
>>> maintaining domains, and qmailadmin works fine for maintaining users.
>>> (Of course, quotas are still broken, so I just use NOQUOTA for all
>>> accounts). Keep in mind though, I'm not an ISP. I help people host
>>> their own.
>
>
Eric Shubert
2011-10-07 14:21:17 UTC
Permalink
On 10/06/2011 07:28 PM, Casey wrote:
>
> I suppose for the large customer I could do something like that, but
> that just one more server to maintain and worry about fixing when
> something goes wrong.

I know. Virtualization makes it less of a hardware requirement, but
there's still the host admin overhead. None the less, doing so brings
all of the global configuration settings down to the domain level, which
can be a plus as well.

> I'll do some research and see if there is
> something out there I'm missing that could accomplish what it is that
> I'm hoping to accomplish.

Be sure to let us know if you find anything.

> What is the quickest/easiest/most efficient way of "freezing" account
> activity? By this I'm assuming you mean, creating new accounts, changing
> passwords, etc?

Send a message to all the email admins?

Comment out "Alias /qmailadmin /usr/share/quamiladmin" line or change
the first "Allow from all" to "Allow from none" in
/etc/httpd/conf/toaster.conf file?

> Casey
>
> Smile Global Technical Support
> Submit or check trouble tickets http://billing.smileglobal.com
> www.smileglobal.com <http://www.smileglobal.com>
>


--
-Eric 'shubes'


---------------------------------------------------------------------------------
Qmailtoaster is sponsored by Vickers Consulting Group (www.vickersconsulting.com)
Vickers Consulting Group offers Qmailtoaster support and installations.
If you need professional help with your setup, contact them today!
Casey
2011-10-08 01:29:30 UTC
Permalink
Thanks for the tips. I'd love to do the virtualization, and hopefully at
some point down the road I'll be able to, but for now I got my hands
full setting up this initial server.

That seems easy enough on disabling qmailadmin...I guess I was
overlooking the easiest possible way of doing it ;-)

I must say - one of the few nice things about creating these users one
domain at a time, is that it is giving me a chance at doing some
inventory, and finding either customers that aren't in the new billing
system and therefore haven't been getting billed, or domains that are no
longer being used. Though I'm not going to lie, its still a PITA.

I actually stumbled across a couple of pieces of software similar to the
two that PV Anthony posted a link to. In a small environment that would
probably be the way to go, but certainly not so much on a larger scale.
Maybe if you could batch copy users/domains...but even at that, how
efficient is it to use IMAP in program/script like that versus simply
copying the maildirs. I'm willing to bet that it would be quicker to
just copy the files, but hey like I said, in a small environment it
might be a nice way to go -- especially if the source and destination
IMAP mailboxes were on two different servers that you didn't necessarily
have admin access to.

Casey

Smile Global Technical Support
Submit or check trouble tickets http://billing.smileglobal.com
www.smileglobal.com <http://www.smileglobal.com>

On 10/7/11 7:21 AM, Eric Shubert wrote:
> On 10/06/2011 07:28 PM, Casey wrote:
>>
>> I suppose for the large customer I could do something like that, but
>> that just one more server to maintain and worry about fixing when
>> something goes wrong.
>
> I know. Virtualization makes it less of a hardware requirement, but
> there's still the host admin overhead. None the less, doing so brings
> all of the global configuration settings down to the domain level,
> which can be a plus as well.
>
>> I'll do some research and see if there is
>> something out there I'm missing that could accomplish what it is that
>> I'm hoping to accomplish.
>
> Be sure to let us know if you find anything.
>
>> What is the quickest/easiest/most efficient way of "freezing" account
>> activity? By this I'm assuming you mean, creating new accounts, changing
>> passwords, etc?
>
> Send a message to all the email admins?
>
> Comment out "Alias /qmailadmin /usr/share/quamiladmin" line or change
> the first "Allow from all" to "Allow from none" in
> /etc/httpd/conf/toaster.conf file?
>
>> Casey
>>
>> Smile Global Technical Support
>> Submit or check trouble tickets http://billing.smileglobal.com
>> www.smileglobal.com <http://www.smileglobal.com>
>>
>
>
Eric Shubert
2011-10-08 02:16:54 UTC
Permalink
On 10/07/2011 06:29 PM, Casey wrote:
> I actually stumbled across a couple of pieces of software similar to the
> two that PV Anthony posted a link to. In a small environment that would
> probably be the way to go, but certainly not so much on a larger scale.
> Maybe if you could batch copy users/domains...but even at that, how
> efficient is it to use IMAP in program/script like that versus simply
> copying the maildirs. I'm willing to bet that it would be quicker to
> just copy the files, but hey like I said, in a small environment it
> might be a nice way to go -- especially if the source and destination
> IMAP mailboxes were on two different servers that you didn't necessarily
> have admin access to.

Keep in mind that all servers don't use Maildir format. If you're coming
from a platform that uses mbox or some other storage format, using imap
for migration can be quite handy.

--
-Eric 'shubes'


---------------------------------------------------------------------------------
Qmailtoaster is sponsored by Vickers Consulting Group (www.vickersconsulting.com)
Vickers Consulting Group offers Qmailtoaster support and installations.
If you need professional help with your setup, contact them today!
Casey
2011-10-08 03:07:36 UTC
Permalink
Ahh that is a good point.

Does QMT use Maildir++ or Maildir?

Oh and another sub-topic for this migration...webmail clients - what is
everyone else out there using? On our old server we provide the
following webmail clients for customers:

TWIG, IMP, SQwebmail, Squirrelmail, and Roundcube

So naturally I'm going to need to get all of these setup on the new
server (luckily for me, Squirrelmail and Roundcube come as toaster's in
QMT :-) and the others shouldn't be too tough). I'm wondering if anyone
knows of or has found any better webmail clients out there that will
work with QMT? I was looking on the wiki and investigated a few of
them...the one that really stuck out was Nutsmail -- but is that really
just more of a premium "skinned" version of Squirrelmail, or are there
features included that make it worth the price? I've also used
Crystalmail a bit in the past, but that's really just a Roundcube clone.

The webmail client is the one piece to the puzzle that may not seem like
a huge deal, but its the single interface that customers have direct
contact with (unless they are using a software client), so it makes
sense for someone such as myself that is providing email service to want
a webmail client that the customers like and find appealing.

Thanks in advance.

Casey

Smile Global Technical Support
Submit or check trouble tickets http://billing.smileglobal.com
www.smileglobal.com <http://www.smileglobal.com>

On 10/7/11 7:16 PM, Eric Shubert wrote:
> On 10/07/2011 06:29 PM, Casey wrote:
>> I actually stumbled across a couple of pieces of software similar to the
>> two that PV Anthony posted a link to. In a small environment that would
>> probably be the way to go, but certainly not so much on a larger scale.
>> Maybe if you could batch copy users/domains...but even at that, how
>> efficient is it to use IMAP in program/script like that versus simply
>> copying the maildirs. I'm willing to bet that it would be quicker to
>> just copy the files, but hey like I said, in a small environment it
>> might be a nice way to go -- especially if the source and destination
>> IMAP mailboxes were on two different servers that you didn't necessarily
>> have admin access to.
>
> Keep in mind that all servers don't use Maildir format. If you're
> coming from a platform that uses mbox or some other storage format,
> using imap for migration can be quite handy.
>
Eric Shubert
2011-10-08 14:54:09 UTC
Permalink
On 10/07/2011 08:07 PM, Casey wrote:
> Ahh that is a good point.
>
> Does QMT use Maildir++ or Maildir?

# rpm -qi qmail-toaster | grep maildir

> Oh and another sub-topic for this migration...

Can we please start a new thread for new topics? This one's getting too
big. ;)

> webmail clients - what is
> everyone else out there using? On our old server we provide the
> following webmail clients for customers:
>
> TWIG, IMP, SQwebmail, Squirrelmail, and Roundcube

Roundcube had security problems early on so its use was discouraged. I
don't know for sure what it's status is. If someone's using it, they
might chime in.

I know there are a couple horde users. With the help of a few others, I
initiated a project to create a horde-toaster package. I need to get
that going again.

Personally, I'd like to see small sub-groups of QMT people who help out
with whichever webmail product(s) they prefer. The way I see it, all of
them can complement QMT, although they're not integral to QMT itself. I
see them more as client software than server software, as that is their
role regarding email.

> So naturally I'm going to need to get all of these setup on the new
> server (luckily for me, Squirrelmail and Roundcube come as toaster's in
> QMT :-)

I would give strong consideration to running webmail(s) on a separate
host. Makes management and load balancing much easier. Not that they
can't be run on the same host though.

> and the others shouldn't be too tough). I'm wondering if anyone
> knows of or has found any better webmail clients out there that will
> work with QMT?

I've heard that eGroupware is nice. Anything that does IMAP should work,
and I expect they all do. Better is a matter of opinion.

> I was looking on the wiki and investigated a few of
> them...the one that really stuck out was Nutsmail -- but is that really
> just more of a premium "skinned" version of Squirrelmail, or are there
> features included that make it worth the price? I've also used
> Crystalmail a bit in the past, but that's really just a Roundcube clone.
>
> The webmail client is the one piece to the puzzle that may not seem like
> a huge deal, but its the single interface that customers have direct
> contact with (unless they are using a software client), so it makes
> sense for someone such as myself that is providing email service to want
> a webmail client that the customers like and find appealing.

And one that can easily be re-branded, and tailored to fit in with other
web apps. Same could probably be said for qmailadmin.

> Thanks in advance.
>
> Casey
>
> Smile Global Technical Support
> Submit or check trouble tickets http://billing.smileglobal.com
> www.smileglobal.com <http://www.smileglobal.com>
>

--
-Eric 'shubes'


---------------------------------------------------------------------------------
Qmailtoaster is sponsored by Vickers Consulting Group (www.vickersconsulting.com)
Vickers Consulting Group offers Qmailtoaster support and installations.
If you need professional help with your setup, contact them today!
Casey
2011-10-08 21:49:44 UTC
Permalink
Hey guys -- I started a new thread for my QMR migration, specifically
for issues/problems encountered...

I'm not sure if this is another flaw with vqadmin or by design,
regardless its yet another reason to check out QControl...

I just migrated a domain that has 165 mailboxes, 159 forwards, and 35
mailing lists. The first problem I stumbled upon was when I was checking
the domain in vqadmin to see that all the users had been added from the
script I ran, it was missing the very last account that was added, and
the 2nd to last account is missing the following information from
vqadmin: "forward flag, vacation flag, quota, administrator, and last
login".

So I was a little concerned that maybe the last two users weren't added
correctly to the system, and this prompted me to check it out with
qmailadmin as well as vuserinfo. Both accounts appeared fine in both
places. I'm wondering if there is a limit to the number of users that
vqadmin will display, or if this is another bug, or if its even worth
writing about.

Casey Price
<http://www.smileglobal.com>
Eric Shubert
2011-10-08 22:19:05 UTC
Permalink
On 10/08/2011 02:49 PM, Casey wrote:
> Hey guys -- I started a new thread for my QMR migration, specifically
> for issues/problems encountered...
>
> I'm not sure if this is another flaw with vqadmin or by design,
> regardless its yet another reason to check out QControl...
>
> I just migrated a domain that has 165 mailboxes, 159 forwards, and 35
> mailing lists. The first problem I stumbled upon was when I was checking
> the domain in vqadmin to see that all the users had been added from the
> script I ran, it was missing the very last account that was added, and
> the 2nd to last account is missing the following information from
> vqadmin: "forward flag, vacation flag, quota, administrator, and last
> login".
>
> So I was a little concerned that maybe the last two users weren't added
> correctly to the system, and this prompted me to check it out with
> qmailadmin as well as vuserinfo. Both accounts appeared fine in both
> places. I'm wondering if there is a limit to the number of users that
> vqadmin will display, or if this is another bug, or if its even worth
> writing about.
>
> Casey Price
> <http://www.smileglobal.com>

I wouldn't wonder. We won't be doing anything but upgrading it at some
point (hopefully soon, along with vpopmail), then see how it behaves.
Thanks for pointing out something to check though.

--
-Eric 'shubes'


---------------------------------------------------------------------------------
Qmailtoaster is sponsored by Vickers Consulting Group (www.vickersconsulting.com)
Vickers Consulting Group offers Qmailtoaster support and installations.
If you need professional help with your setup, contact them today!
Casey
2011-10-08 22:31:46 UTC
Permalink
No prob. What kind of a timeline are you guys looking at for the next
set of upgrades in QMT? Is it something far enough out that I shouldn't
worry about it until after I complete my migration and have the new
server in place? I'd like to help out however I can :-)

Casey

Smile Global Technical Support
Submit or check trouble tickets http://billing.smileglobal.com
www.smileglobal.com <http://www.smileglobal.com>

On 10/8/11 3:19 PM, Eric Shubert wrote:
> On 10/08/2011 02:49 PM, Casey wrote:
>> Hey guys -- I started a new thread for my QMR migration, specifically
>> for issues/problems encountered...
>>
>> I'm not sure if this is another flaw with vqadmin or by design,
>> regardless its yet another reason to check out QControl...
>>
>> I just migrated a domain that has 165 mailboxes, 159 forwards, and 35
>> mailing lists. The first problem I stumbled upon was when I was checking
>> the domain in vqadmin to see that all the users had been added from the
>> script I ran, it was missing the very last account that was added, and
>> the 2nd to last account is missing the following information from
>> vqadmin: "forward flag, vacation flag, quota, administrator, and last
>> login".
>>
>> So I was a little concerned that maybe the last two users weren't added
>> correctly to the system, and this prompted me to check it out with
>> qmailadmin as well as vuserinfo. Both accounts appeared fine in both
>> places. I'm wondering if there is a limit to the number of users that
>> vqadmin will display, or if this is another bug, or if its even worth
>> writing about.
>>
>> Casey Price
>> <http://www.smileglobal.com>
>
> I wouldn't wonder. We won't be doing anything but upgrading it at some
> point (hopefully soon, along with vpopmail), then see how it behaves.
> Thanks for pointing out something to check though.
>
Eric Shubert
2011-10-08 22:52:36 UTC
Permalink
On 10/08/2011 03:31 PM, Casey wrote:
> No prob. What kind of a timeline are you guys looking at for the next
> set of upgrades in QMT? Is it something far enough out that I shouldn't
> worry about it until after I complete my migration and have the new
> server in place? I'd like to help out however I can :-)
>
> Casey
>
> Smile Global Technical Support
> Submit or check trouble tickets http://billing.smileglobal.com
> www.smileglobal.com <http://www.smileglobal.com>
>

Yes, and thanks.
The timeline we're looking at is when it gets done. There's no schedule.
--
-Eric 'shubes'


---------------------------------------------------------------------------------
Qmailtoaster is sponsored by Vickers Consulting Group (www.vickersconsulting.com)
Vickers Consulting Group offers Qmailtoaster support and installations.
If you need professional help with your setup, contact them today!
Jake Vickers
2011-10-10 13:53:36 UTC
Permalink
On 10/08/2011 06:52 PM, Eric Shubert wrote:
> On 10/08/2011 03:31 PM, Casey wrote:
>> No prob. What kind of a timeline are you guys looking at for the next
>> set of upgrades in QMT? Is it something far enough out that I shouldn't
>> worry about it until after I complete my migration and have the new
>> server in place? I'd like to help out however I can :-)
>>
>> Casey
>>
>> Smile Global Technical Support
>> Submit or check trouble tickets http://billing.smileglobal.com
>> www.smileglobal.com <http://www.smileglobal.com>
>>
>
> Yes, and thanks.
> The timeline we're looking at is when it gets done. There's no schedule.


Thanks Eric.
The hold up on this is me, and my time is limited these days. Between
job/shift changes, moving, remodeling, etc., I haven't invested a lot of
time to the project over the last year. That whole life comes at you
fast thing. :)
I have a couple of packages up for testing that I announced to the
development list, and they're currently being tested. There's a little
more work to do, and then some stress testing.

---------------------------------------------------------------------------------
Qmailtoaster is sponsored by Vickers Consulting Group (www.vickersconsulting.com)
Vickers Consulting Group offers Qmailtoaster support and installations.
If you need professional help with your setup, contact them today!
Casey
2011-10-04 19:02:09 UTC
Permalink
Ok that sounds easy enough ;-)

I'll read up on Dovecot.

From the sounds of it, I may be able to take my current migrate-domain
script and make some modifications to it, as it already creates a
tarball of each mailbox the whatever domain you specify, and then it
generates a script that recreates the account on the new machine and
restores the tarball to the right place on the new server. So I could
take out the piece that creates the account, and instead just use the
portion that restores the maildir. That part shouldn't be too incredibly
difficult, however what I'm a little unsure of is how to automate the
process - consider this example:

Say I ran through the migration and got every single account migrated
today and was going to make the transition over the next day or two - I
could make the changes in DNS, and then change smtproutes, but how would
I make certain that I had all of the latest data migrated to the new
system? I mean, I guess like you said i could stop the pop/imap services
on the old server and then back up each mailbox, and restore on the new
server...BUT...I don't understand how I could restore the mail from the
old server to the new server without potentially overwriting new mail or
causing duplicate mail from what they had previously sent, received,
deleted, etc. Essentially I'd be doing the same work twice (well maybe
not exactly the same work, because the 2nd time around data would just
be getting restored), but it still would mean going through each domain,
and well...copying 165GB of mostly the same data again.

Am I missing something here?

/"The only problem I can think of is if you have pop users who leave
messages on the server. Those are going to be duplicated. If your users
can't live with that, you'll need to do some extra work to get that part
taken care of."
/ -- How might I accomplish this part?

Thanks again,

Casey

Smile Global Technical Support
Submit or check trouble tickets http://billing.smileglobal.com
www.smileglobal.com <http://www.smileglobal.com>

On 10/4/11 10:29 AM, Eric Shubert wrote:
> On 10/04/2011 09:42 AM, ***@smileglobal.com wrote:
>> For now no, the gateway and sa hosts will stay put.
>
> Then you can simply change the smtproutes in the sa host to point to
> the new server at an appropriate time, after the new host is up and
> running and accounts exist there.
>
>> Understandable about keeping the old server up while changes propagate.
>
> Don't need to worry about the inbound side though, as that'll stay the
> same.
>
>> The old server is using courier, but I guess dovecot could be a better
>> route. I've used it a little in the past on a different server. What
>> kind of difficulty am I looking at to switch from courier to dovecot?
>
> It's not tough at all. See the wiki for instructions. There's an rpm
> for QMT available in the QTP repo.
>
>> Also, how would you recommend that I keep the maildirs up to date? Im
>> trying to figure out the best way to do that without causing either lost
>> or duplicate mail. Am I going to want to create tarballs of mailboxes
>> and run some type of conversion on them before restoring them, or
>> utilize sync, or is there a better way?
>
> I think you're gonna need to do some scripting for this, as the
> directory structure is changing. Basically, you want something that'll
> migrate the maildir folders from one host to another. I think I'd use
> tar with a couple scripts to 'drive' the thing.
>
> On the old side, the script should find a domain (or drive it with a
> list), then find each account, then tarball up the maildir into a
> single file. At the end of the process, you can tarball up all the
> account tarballs into a single archive for transfer.
>
> After transfer, on the new side, untar the transferred file, and
> another script would process each maildir tarball, and untar it into
> the appropriate place. Be careful not to wipe out any mail that may
> have come in directly to the new server already.
>
> The only problem I can think of is if you have pop users who leave
> messages on the server. Those are going to be duplicated. If your
> users can't live with that, you'll need to do some extra work to get
> that part taken care of.
>
>> Thanks for all the help you've been offering...I certainly appreciate
>> it.
>
> Sure.
> I hope you're taking copious notes for the wiki. ;)
>
Eric Shubert
2011-09-27 01:56:08 UTC
Permalink
On 09/26/2011 04:19 PM, Casey wrote:
> the major problem was that vqadmin would show the domain, but wouldn't
> recognize that there were accounts associated with it.

vqadmin has some bugs in it that might get straightened out when we
upgrade to vpopmail-5.33. In the meantime, I wouldn't count on vqadmin
for much of anything. Even though vqadmin isn't showing everything, you
might have a viable install. Does qmailadmin show the accounts? If the
CLI tools show that everything's ok and qmailadmin is working ok, then I
expect your install might be ok.

--
-Eric 'shubes'


---------------------------------------------------------------------------------
Qmailtoaster is sponsored by Vickers Consulting Group (www.vickersconsulting.com)
Vickers Consulting Group offers Qmailtoaster support and installations.
If you need professional help with your setup, contact them today!
Casey
2011-09-27 02:36:17 UTC
Permalink
Turns out, all I had to do was recompile vqadmin after I had recompiled
vpopmail. Then my next problem was that I couldn't log into the account
using squirrelmail, so to resolve that I recompiled courier-imap, and
courier-authlib.

I think I'll go with Jake's suggestion and try to standardize everything
as I migrate it that way I won't have to do the same work twice.
Casey

Smile Global Technical Support
Submit or check trouble tickets http://billing.smileglobal.com
www.smileglobal.com <http://www.smileglobal.com>

On 9/26/11 6:56 PM, Eric Shubert wrote:
> On 09/26/2011 04:19 PM, Casey wrote:
>> the major problem was that vqadmin would show the domain, but wouldn't
>> recognize that there were accounts associated with it.
>
> vqadmin has some bugs in it that might get straightened out when we
> upgrade to vpopmail-5.33. In the meantime, I wouldn't count on vqadmin
> for much of anything. Even though vqadmin isn't showing everything,
> you might have a viable install. Does qmailadmin show the accounts? If
> the CLI tools show that everything's ok and qmailadmin is working ok,
> then I expect your install might be ok.
>
Eric Shubert
2011-09-27 03:08:01 UTC
Permalink
Good advice that.

BTW, those packages are dependent on the vpopmail libs, which is why
they need to be rebuilt whenever vpopmail changes (now that I think of it).

--
-Eric 'shubes'

On 09/26/2011 07:36 PM, Casey wrote:
> Turns out, all I had to do was recompile vqadmin after I had recompiled
> vpopmail. Then my next problem was that I couldn't log into the account
> using squirrelmail, so to resolve that I recompiled courier-imap, and
> courier-authlib.
>
> I think I'll go with Jake's suggestion and try to standardize everything
> as I migrate it that way I won't have to do the same work twice.
> Casey
>
> Smile Global Technical Support
> Submit or check trouble tickets http://billing.smileglobal.com
> www.smileglobal.com <http://www.smileglobal.com>
>
> On 9/26/11 6:56 PM, Eric Shubert wrote:
>> On 09/26/2011 04:19 PM, Casey wrote:
>>> the major problem was that vqadmin would show the domain, but wouldn't
>>> recognize that there were accounts associated with it.
>>
>> vqadmin has some bugs in it that might get straightened out when we
>> upgrade to vpopmail-5.33. In the meantime, I wouldn't count on vqadmin
>> for much of anything. Even though vqadmin isn't showing everything,
>> you might have a viable install. Does qmailadmin show the accounts? If
>> the CLI tools show that everything's ok and qmailadmin is working ok,
>> then I expect your install might be ok.
>>



---------------------------------------------------------------------------------
Qmailtoaster is sponsored by Vickers Consulting Group (www.vickersconsulting.com)
Vickers Consulting Group offers Qmailtoaster support and installations.
If you need professional help with your setup, contact them today!
Casey
2011-09-27 03:28:41 UTC
Permalink
Definitely good to know now ;-)

Do you have a list of all of the packages that are should be rebuilt
when vpopmail is rebuilt? So if I do the manual process of re-adding
users with a script, what would be the easiest way to make sure that the
quotas and other properties associated with the account are re-created?

Thanks,

Casey

Smile Global Technical Support
Submit or check trouble tickets http://billing.smileglobal.com
www.smileglobal.com <http://www.smileglobal.com>

On 9/26/11 8:08 PM, Eric Shubert wrote:
> Good advice that.
>
> BTW, those packages are dependent on the vpopmail libs, which is why
> they need to be rebuilt whenever vpopmail changes (now that I think of
> it).
>
Pak Ogah
2011-09-27 04:21:35 UTC
Permalink
On 09/27/11 6:19, Casey wrote:
> Thanks Jake. I'm hoping to make this process as automated as possible...
>
> What I'm wondering, is does the database need to be converted somehow
> to work properly on the new server since the old server is running
> mysql 4 and the new one is mysql 5?
I can't answer others, but for Mysql database which vpopmail used you
can easily converted database ver 4 into ver 5. here's how I do it:
- make sure you have backup for vpopmail database
- stop mysql service on both servers
- copy vpopmail database from /var/lib/mysql/vpopmail from old server
into new server
- start mysql service on new server
- then run mysql_upgrade with your privileges
see mysql_upgrade --help and
http://dev.mysql.com/doc/refman/5.0/en/mysql-upgrade.html for detail option
afaik vpopmail database is using myisam, if your vpopmail database on
your old server (solaris) not using myisam, you can export the database
into *.sql file using mysqldump then import it to new server. see
http://dev.mysql.com/doc/refman/5.1/en/mysqldump.html
Loading...