Partition switch with unaligned partition

When I switch a partition off a partitioned table with unaligned indexes, I first drop the unaligned indexes, then switch the partition, and lastly recreate the unaligned index.  That has worked wonderfully, except for how long it takes to recreat